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DISCLAIMER 


The  views,  opinions ,  and/or  findings  contained  in  this  report  are  those  of 
the  author (s)  and  should  not  be  construed  as  an  official  Department  of  the 
Army  position ,  policy ,  or  decision  unless  so  designated  by  other  official 
documentation. 


WARNING 


Information  and  data  contained  herein  are  based  on  the  input  available  at  the 
time  of  preparation.  The  results  are  subject  to  change . 


EXECUTIVE  SUMMARY 


This  report  presents  the  results  of  the  Phase  I  (the  first  scoring  year) 
evaluation  of  the  three-year  test  implementation  of  the  US  Army  TRADOC  Systems 
Analysis  Activity  (TRASANA)  Technical  Staff  Productivity  Measurement  System 
(TPMS). 

TPMS  is  a  methodology  designed  to  measure  the  productivity  of  approxi¬ 
mately  250  technical  staff  members  who  are  mostly  scientists  and  engineers 
engaged  in  various  facets  of  operations  research  and  applied  problem  solving. 
Most  of  the  work  is  performed  by  teams  organized  to  meet  the  requirements  of 
each  project.  The  organization  completes  60  or  more  projects  a  year  which  are 
quite  diverse  in  nature.  Each  project  is  required  by  TPMS  to  have  a  project 
plan ,  which  is  a  detailed  planning  document  including  estimates  of  manpower 
and  computer  resources  needed  for  the  project,  as  well  as  a  time  schedule  for 
the  project. 

The  project  plan  is  submitted  for  approval  to  the  Productivity  Measure¬ 
ment  Board  (PMB)  which  is  responsible  for  monitoring  resource  estimates  and 
schedules.  Once  the  plan  is  approved,  the  team  executes  it.  When  the  project 
is  completed,  the  project  is  assigned  scores  based  on  how  well  the  actual 
resource  utilization  and  time  schedule  agreed  with  the  plan,  as  well  as  a 
product  quality  score.  The  product  quality  score  is  assigned  by  a  quality 
control  group  called  the  Product  Review  Board  (PRB).  The  scores  for  all  of 
the  factors  are  multiplied  together  to  arrive  at  the  Project  Score  (PS).  Two 
or  more  PS's  can  be  combined  and  weighted  by  expended  manpower  to  form  the 
Productivity  Index  (PI).  Pi's  are  calculated  for  each  individual  and  for 
organizational  units  of  TRASANA. 

TPMS  was  developed  at  TRASANA  and  adopted  for  a  three  year  trial  imple¬ 
mentation  starting  1  Oct  80.  The  first  scoring  period  was  a  truncated  year 
(nine  months)  ending  30  Jun  81  to  put  the  system  on  the  1  July  to  30  June 
year. 


The  objectives  of  this  study  were  (1)  to  determine  how  the  TRASANA 
technical  staff  (TTS)  perceives  TPMS,  (2)  to  determine  the  degree  to  which 
TPMS  is  a  meaningful  measure  of  productivity,  and  (3)  to  develop  a  TPMS  data 
base  consisting  of  Phase  I  TPMS-T  results. 

The  first  study  objective  was  answered  by  surveying  the  TTS  at  the  end  of 
the  first  year.  The  second  objective  ixis  answered  by  an  in-depth  analysis  of 
the  first  year  TPMS  scoring  results,  a  mathematical  analysis  of  the  scoring 
algorithm,  a  functional  analysis  of  the  PMB  and  PRB,  and  by  a  PI  validation 
analysis.  The  PI  validation  consisted  of  developing  a  measure  of  producti¬ 
vity  which  was  independent  of  TPMS,  and  which  could  be  used  for  comparison 
with  the  PI.  This  independent  measure,  or  criterion  measure  (CM),  consisted 
of  confidential  supervisory  ratings  obtained  by  personal  interview.  The  third 
,  objective  was  achieved  by  forming  a  computer  data  file  consisting  of  both  the 
survey  and  first  year  scoring  results. 
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The  major  findings  were: 


0  The  staff  reaction  to  TPMS  was  mixed.  They  see  TPMS  as  a 

potentially  useful  managerial  tool,  but  there  is  a  high  level  of 
concern  about  PI  inaccuracies,  system  manipulation,  and  the 
effect  on  the  creative  and  cooperative  atmosphere  at  TRASANA . 

0  The  system  was  not  seen  as  fair  by  the  staff. 

0  TPMS  reflects  organizational  productivity  in  the  classical  sense 
( value  of  output  over  value  of  input,  and  captures  both 
efficiency  and  effectiveness)  but  its  accuracy  needs  to  be 
improved. 

°  The  resource  estimates,  especially  manpower,  were  not  adequately 
controlled  by  the  PMB,  which  decreased  the  accuracy  of  the 
system. 

0  The  individual  PI  did  not  accurately  reflect  individual 
productivity . 

0  There  is  considerable  potential  for  input  effort/output  score 
mismatch  with  the  individual  PI. 

0  The  scoring  algorithm,  especially  the  effect  of  judgmental  and 
discretionary  awards,  was  not  well  understood. 

°  Organizational  units  with  an  overhead  mission  are  disadvantaged 
by  including  overhead  (activities  that  are  not  project  specific) 
in  the  PI  computation. 

0  The  organizational  PI  was  more  accurate  than  the  individual  PI, 
but  it  will  be  much  more  accurate  if  the  resource  estimates  are 
controlled  and  overhead  is  more  consistently  reported  across 
organizational  units. 

0  The  PRB  serves  a  valuable  quality  control  function,  but  the 
quality  scoring  was  much  less  successful. 

The  major  conclusions  were: 

°  TRASANA  is  a  healthy  organization  with  high  morale  and  a 
motivated  work  force. 

0  TPMS  is  a  conceptually  sound  productivity  measurement  system,  but 
many  of  its  functional  elements  need  revision. 

0  The  individual  PI  should  be  eliminated. 

0  The  organizational  PI  would  be  strengthened  by  eliminating 
overhead  from  the  PI  calculation. 

0  A  new  method  of  controlling  resource  estimates  is  required. 


°  The  seo vino  algorithm,  is  unnecessarily  complicated. 

°  Firm  guidelines  are  needed  for  overhead  accounting . 

Major  recommendations  included: 

°  Use  only  three  scoring  factors  -  manpower,  timeliness  and  quality 
-  and  score  all  three  by  categories,  c.g.,  high,  medium,  and 
low.  The  three  category  scores  for  manpower  and  for  timeliness 
should  reward  accuracy  in  planning. 

°  Change  the  125 %  rule  (percentage  of  available  manpower  committed 
to  projects)  to  the  110%  rule. 

°  Institute  an  after-the-fact  division  level  check  with  reward  for 
accuracy  in  estimating  manpower  and  timeliness. 

°  Eliminate  the  individual  PI. 

°  Establish  overhead  changing  guidelines  with  a  specified  percentage 
of  manpower  allowed  to  be  charged  to  overhead  without  penalty. 

°  Eliminate  overhead  from  organizational  PI  computation. 

°  Establish,  a  scoring  checklist  (with  scoring  criteria)  to  be  used 
by  the  PRB  for  scoring  each  product. 

°  Assign  a  productivity  rating  to  the  PRB. 

°  Maintain  management  intervention  to  avoid  penalties  in  PI  from 
undertaking  risky  projects . 

An  alternative  TPMS  scoring  procedure  incorporating  the  above  recommenda¬ 
tions  is  included  in  the  report. 
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PHASE  I  EVALUATION  OF  TRASANA  TECHNICAL  STAFF 
PRODUCTIVITY  MEASUREMENT  SYSTEM-TEST  (TPMS-TE) 

CHAPTER  1 

INTRODUCTION 


1.1  PURPOSE 

This  report  presents  the  results  of  the  Phase  I  (the  first  scoring  year) 
evaluation  of  the  three-year  test  implementation  of  the  US  Army  TRADOC  Systems 
Analysis  Activity  (TRASANA)  Technical  Staff  Productivity  Measurement  System 
(TPMS). 

1.2  BACKGROUND 

a.  TPMS  is  a  methodology  designed  to  measure  the  productivity  of  the 
TRASANA  technical  staff  (TTS),  its  organizational  elements,  and  its  individual 
employees.  It  yields  a  single  dimensionless  index  as  the  measure  of  produc¬ 
tivity.  The  formulation  of  this  index  is  intended  to  be  consistent  with  the 
traditional  definition  of  productivity  as  value  of  output  divided  by  value  of 
input.  In  principle,  TPMS  places  TRASANA  on  a  "fixed  price  plus  incentive" 
basis  for  producing  its  various  products.  TPMS  was  designed  with  the 
following  attributes  in  mind: 

(1)  Relevant  -  to  the  organization,  its  products,  and  to  the  jobs 
that  produce  the  products. 

(2)  Rational  and  fair  -  so  that  it  can  be  accepted  and  used  by  a 
sophisticated  workforce. 

(3)  Affordable  -  in  terms  of  direct  and  indirect/overhead  costs  for 
operation  and  maintenance. 

(4)  Flexible  -  to  couple  with  a  dynamic  workload,  organization,  and 
operations. 

(5)  Simple  -  both  in  principle  and  practice. 

(6)  Transparent  -  so  as  to  facilitate  understanding  of  cause  and 
effect  relationships. 

(7)  Durable  -  to  support  multi-year  use. 

b.  TPMS  was  adopted  at  TRASANA  for  a  three  year  trial  beginning  1  Oct 
80.  The  first  scoring  period  ended  30  Jun  81,  and  was  nine  months  in  length. 
Subsequent  years  will  run  from  1  July  through  30  June.  Management  formed  a 
small  internal  evaluation  team  in  Mar  81,  and  charged  the  team  with  developing 
and  executing  an  objective  evaluation  of  the  TPMS  test  implementation.  The 
study  plan  developed  for  evaluating  the  initial  implementation  period  is  in 
appendix  A.  In  keeping  with  management's  desire  to  have  as  independent  and 
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objective  evaluation  as  possible,  the  study  plan  was  developed  independently 
by  the  study  team,  and  reviewed,  but  not  changed,  by  management.  The  evalua¬ 
tion  effort  was  directed  toward  determining  the  extent  to  which  TPMS  met  its 
design  attributes. 

1.3  THE  SYSTEM 

A  complete  description  of  TPMS,  its  development  and  the  three  year  trial 
implementation  period  can  be  found  in  other  documents. ?  This  brief  overview 
is  provided  for  the  reader  who  is  not  familiar  with  the  system. 

a.  TRASANA  has  nearly  250  personnel  who  are  referred  to  as  analysts  or 
technical  staff.  They  are  mostly  scientists  and  engineers  who  engage  in 
various  facets  of  operations  research  and  applied  problem  solving.  The  modus 
operandi  is  for  a  team  to  be  assigned  to  a  given  project.  The  team  is  respon¬ 
sible  for  clearly  defining  the  problem,  developing  and  executing  an  appropri¬ 
ate  methodol  ogy  ,  and  publishing  a  report  documenting  the  results  of  the 
effort. 

b.  TPMS  basically  provides  a  method  for  assigning  a  numerical  score  to 
each  project  after  it  is  completed.  This  score  is  called  the  project  score 
(PS).  The  PS  is  converted  to  a  weighted  index  (weighted  by  invested  resources) 
which  can  in  turn  be  converted  to  a  weighted  productivity  index  (PI)  for 
individuals  as  well  as  organizational  units. 

c.  The  mechanics  of  the  system  are  rather  straight  forward.  When  a 
project  is  identified,  a  senior  analyst  is  assigned  as  the  study  manager  (SM). 
The  SM  is  responsible  for  negotiating  details  of  the  project  with  any  outside 
agencies  and  writing  a  project  plan.  The  project  plan  is  a  detailed  planning 
document  which  breaks  down  the  project  into  tasks  and  subtasks,  specifies  the 
manpower  and  computer  resources  needed  for  each  subtask,  and  also  specif ier 
what  the  product  of  each  subtask  will  be  and  when  it  will  be  completed.  Once 
this  document  is  completed,  it  is  submitted  to  the  Productivity  Measurement 
Board  (PMB),  which  consists  of  the  Assistant  Deputy  for  Technical  Operations 
(chairperson)  and  the  eight  functional  division  chiefs.  The  PMB  reviews  each 
project  plan  for  completeness  and  to  insure  the  resource  requirements  and 
completion  dates  are  realistic. 

d.  Once  a  project  plan  is  approved  by  the  PMB,  the  basis  for  assigning  a 
project  score  has  been  established.  There  are  five  factors  contributing  to 
the  PS:  manpower  utilization  efficiency  (R),  computer  utilization  efficiency 
(C),  timeliness  (T),  product  quality  (Q),  and  priority  (P).  These  factors 
each  take  on  a  limited  value  as  shown  in  table  1-1.  The  R  and  C  factor  scores 
are  essentially  a  ratio  of  planned  to  expended  resources.  T  is  based  on  how 


1 TRASANA  Memorandum  5-4,  TRASANA  Technical  Staff  Productivity  Measurement 
System  (TPMS)-Test,  August  1980. 

TRASANA  Technical  Documentation  37-80,  Productivity  Measurement  System  Design 
and  Development,  December  1980. 
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TABLE  1-1.  TPMS  SCORING  FACTOR  VALUES 


FACTOR 

MINIMUM  VALUE 

MAXIMUM  VALUE 

R 

0.7 

1.2 

c 

0.9 

1.1 

T 

0.8 

1.1 

Q 

1.0 

1.5 

P 

1.0 

1.2 

close  the  product  is  completed  to  the  projected  completion  date.  Q  is 
assigned  by  the  Product  Review  Board  (PRB),  which  is  an  internal  quality 
control  mechanism  consisting  of  three  permanent  (voting)  members  augmented  by 
additional  analysts  selected  on  an  ad  hoc  (non-voting)  basis.  P  is  1.2  if  the 
study  is  conducted  for  an  external  customer,  and  is  negotiated  with  the  PMB  in 
other  cases.  The  PS  is  computed  by  multiplying  all  of  the  factor  scores 
together : 


PS  =  R  x  C  x  T  x  Q  x  P 


1.4  PROBLEM 

The  problem  addressed  in  the  Phase  I  evaluation  was  to  determine  the  degree  to 
which  TPMS  provides  a  meaningful  measure  of  the  productivity  of  the  TRASANA 
technical  staff  and  its  organizational  elements. 

1.5  IMPACT  OF  PROBLEM 

TPMS  is  a  new  system.  Numerous  decisions  were  made  in  designing  the  system 
which  involved  trade-offs.  The  only  way  to  know  how  well  the  final  result 
achieves  its  goals  is  to  test  it.  Furthermore,  a  comprehensive  evaluation  of 
the  test  is  required  to  fully  capture  the  effects  of  the  system,  and  to 
document  its  strengths  and  weaknesses.  Failure  to  comprehensively  evaluate 
TPMS  performance  during  the  test  period  could  result  in  implementing  a  system 
which  does  not  adequately  accomplish  its  objective,  or  which  even  may  be 
counterproductive. 

1.6  OBJECTIVES  AND  STUDY  QUESTIONS 

a.  To  determine  how  the  TRASANA  technical  staff  perceives  TPMS.  The 
study  questions  for  this  objective  are: 
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(1)  What  is  the  TTS  general  attitude  toward  TPMS? 

(2)  What  is  the  perceived  motivational  impact  of  TPMS? 

(3)  Is  TPMS  seen  as  a  viable  managerial  tool? 

b.  To  determine  the  degree  to  which  TPMS  is  a  meaningful  measure  of 
productivity.  The  study  questions  for  this  objective  are: 

(1)  To  what  extent  do  TPMS  ratings  agree  with  confidential  super¬ 
visory  productivity  assessments  for  individuals  and  branches? 

(2)  Does  TPMS  have  generally  the  same  impact  on  the  different 
TRASANA  organizational  divisions? 

(3)  What  are  the  functional  relationships  amonq  the  scorino 
algorithm  variables? 

(4)  What  is  the  operational  impact  of  the  Productivity  Measurement 
Board  (PMB)  and  Product  Review  Board  (PRB)? 

c.  To  develop  a  TPMS  data  base  consisting  of  Phase  I  TPMS-T  results. 
Available  data  on  resource  implications  will  be  included.  This  data  base  will 
form  the  benchmark  for  comparison  of  future  results. 

1.7  SCOPE 

a.  The  Phase  I  evaluation  was  limited  to  the  results  of  the  first  trial 
implementation  year  (nine  months)  of  TPMS. 

b.  This  evaluation  was  limited  exclusively  to  TPMS.  While  TPMS  was 
developed  in  part  as  a  local  response  to  requirements  of  the  new  Performance 
Appraisal  (PA)  and  Merit  Pay  (MP)  systems  resulting  from  the  1978  Civil 
Service  Reform  Act,  no  assessment  of  the  PA  or  MP  systems  was  intended  nor 
attempted. 


c.  TPMS-TE  is  an  assessment  of  a  specific  productivity  measurement 
system.  It  is  not  intended  to  be  a  comprehensive  study  of  methods  to  measure 
TRASANA  productivity  or  research  and  development  type  productivity  in  general. 

d.  One  of  the  desired  characteristics  of  TPMS  is  that  it  be  affordable. 
Available  data  will  be  collected  on  resources  expended  on  the  first  year  test 
of  TPMS;  however,  judgments  on  whether  the  costs  are  a  worthwhile  investment 
are  beyond  the  scope  of  this  study. 

1.8  ASSUMPTIONS,  CONSTRAINTS,  AND  LIMITATIONS 

a.  TPMS  will  be  applied  equally  to  all  TRASANA  Technical  Staff  (military 
and  civilian). 

b.  Key  organizational  supervisors  know  the  relative  long  term 
productivity  of  employees  they  supervise. 
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c.  Division  chiefs,  branch  chiefs,  and  the  TTS  provided  complete  and 
accurate  information  for  the  study. 

d.  Not  all  of  the  TTS  worked  on  TPMS  scored  projects  during  the  first 
scoring  period. 

e.  Since  there  was  no  uniform,  quantitative  productivity  measure  prior 
to  TPMS,  there  is  no  basis  for  comparing  test  period  productivity  with  past 
productivity. 

f.  There  is  no  independent  measure  of  productivity  with  which  to 
directly  compare  the  TPMS  results. 


CHAPTER  2 


METHODOLOGY 


2.1  APPROACH 

The  evaluation  of  the  functioning  of  TPMS  during  the  first  trial  scoring 
period  was  approached  in  three  ways  as  depicted  in  figure  2-1.  First,  the 
reaction  of  the  TRASANA  technical  staff  was  assessed.  Data  for  this  assess¬ 
ment  were  collected  by  survey  at  the  end  of  the  first  scoring  period,  but 
before  scoring  results  were  generally  available.  Second,  an  independent 
measure  (criterion)  was  developed  to  determine  the  degree  to  which  the 
productivity  index  (PI)  reflects  productivity.  Third,  the  functioning  of 
selected  TPMS  components  were  evaluated  using  data  collected  from  several 
sources.  The  procedures  utilized  for  each  of  these  analyses  are  presented  in 
the  following  paragraphs. 

2.2  TRASANA  TECHNICAL  STAFF  PERCEPTIONS 
2.2.1  TRASANA  Technical  Staff  (TTS)  Survey 

a.  The  survey  was  developed  to  answer  specific  management  questions, 
as  well  as  to  provide  information  required  by  the  study  team  for  the  evalua¬ 
tion.  The  results  of  an  invited  Organizational  Effectiveness  Visit,  9-12  Feb 
81  indicated  considerable  uneasiness  about  TPMS.T  Management  desired  more 
precise  information  about  the  level  of  concern  among  the  staff,  and  wanted  to 
know  what  particular  aspects  of  TPMS  were  precipitating  the  anxiety.  In 
providing  this  information  for  management,  a  baseline  would  also  be  estab¬ 
lished  which  could  be  used  to  document  any  shifts  that  occurred  over  the 
remaining  two  years  of  the  trial  implementation. 

b.  In  addition  to  staff  reaction  to  TPMS,  the  survey  provided  the 
following  types  of  information: 

°  Respondent  background  (survey  was  anonymous) 

°  Feelings  toward  TRASANA 

°  Job  satisfaction 

0  Understanding  of  TPMS 

c.  After  specific  objectives  for  the  survey  were  defined,  questions 
were  constructed,  and  a  draft  questionnaire  was  printed  and  pilot-tested.  The 
pilot  test  consisted  of  administering  the  survey  to  two  groups  (n  =  5,  6)  of 
TRASANA  analysts  (cross-validation  design).  The  survey  was  given  to  each 


IfRADOC  OE  Team  (ODCS  PAL)  Memorandum  for  Director ,  TRASANA,  24  Feb  81, 
subject:  Organizational  Effectiveness  Visit ,  9-12  Feb  81. 
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analyst  individual ly ,  and  comments  were  solicited  on  clarity,  difficulty  in 
answering  any  questions,  and  whether  the  questions  provided  ample  opportunity 
for  expressing  all  feelings  and  concerns  about  TPMS.  Based  on  these  comments 
and  a  statistical  item-analysis,  revisions  to  the  survey  were  made.  The 
revised  survey  contained  91  items  organized  into  three  parts:  demographics, 
attitude  toward  TRASANA  and  TPMS,  and  TPMS  familiarity.  There  was  also  a 
comment  page  attached  to  the  survey.  A  copy  of  the  survey  is  in  appendix  B. 

2.2.2  Sample 


a.  The  survey  was  administered  to  190  analysts,  or  approximately  80 

percent  of  the  TTS  with  experience  under  TPMS.  Those  who  did  not  take  the 
survey  fell  into  one  or  more  of  several  categories.  The  survey  authors  and 

those  who  took  the  pilot  survey  were  not  considered  part  of  the  available 

population.  Furthermore,  persons  who  were  identified  as  new  to  the  organiza¬ 
tion,  and  therefore  not  familiar  with  either  the  organization  or  TPMS,  were 
also  excluded.  The  majority  of  the  remaining  persons  who  did  not  take  the 
survey  were  unavailable  due  to  a  combination  of  leave  and  work  related  travel 
requi rements. 

b.  Table  2-1  provides  a  comparison  of  the  survey  sample  and  the  TTS  by 

grade.  Examination  of  the  data  reveals  the  relative  homogeneity  of  the  two 

groups.  The  two  distributions  were  compared  statistically  and  found  not  to  be 
different.2  The  sample  is  therefore  considered  statistically  representative 
of  the  TTS  by  grade. 

c.  Table  2-2  shows  the  participation  rate  in  the  survey  by  division. 
Percent  participation  in  the  table  is  not  necessarily  percent  of  total 
assigned  who  took  the  survey,  because  pilot  test  participants,  evaluation  team 
members,  and  identified  new  personnel  were  excluded.  All  division  chiefs  and 
20  branch  chiefs  took  the  survey.  A  statistical  comparison  was  made  between 
the  distribution  of  survey  respondents  by  division,  and  the  actual  assigned 
personnel  by  division  to  determine  if  the  division  representation  in  the 
survey  sample  was  biased. 3  There  was  no  statistically  significant  difference 
in  the  distributions;  therefore,  no  division  was  over  or  under  represented  to 
the  point  of  biasing  the  results. 


2The  test  for  independence  ms  with  the  Chi-square  technique .  X2  =  22.55, 
df  =  8,  Which  is  not  statistically  significant  at  the  p  <  .05  level 
(X2  =  15.51). 

8,  .05 

3This  comparison  was  by  Chi-Square  using  actual  staffing  figures  from  the  Aug 
81  TRASANA  Manpower  Account  Roster.  X 2  =  6.19,  df  =  7,  which  is  not  statis¬ 
tically  significant  at  the  p  <  .05  level  (X2  =  14.07).  Table  C-l  in 

7 ,  .0 5 

appendix  C  shows  the  composition  of  the  TTS  and  the  survey  sample  by  division . 
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TABLE  2-1 


COMPARISON  OF  COMPOSITION  OF  SURVEY 
SAMPLE  AND  TECHNICAL  STAFF  POPULATION  BY  GRADE 


NUMBER 

PERCENT 

PERCENT 

IN 

OF 

OF 

GRADE 

SAMPLE 

SAMPLE 

POPULATION 

CIVILIAN: 

GS-7 

3 

1.6 

4.2 

9 

9 

4.7 

3.5 

11 

14 

7.4 

6. 5 

12 

51 

26.8 

25.4 

13 

53 

27.9 

26.2 

14 

27 

14.2 

12.3 

15 

8 

4.2 

3.5 

Other 

3 

1.6 

-- 

MILITARY: 

Officer 

16 

8.4 

12.7 

Enl  i  sted 

6 

3.2 

5.8 

TOTAL 

190 

100 

a 

rH 

• 

o 

o 

aSouvce:  TRASANA  Manpower  Management  Branch,  August  1981  Manpower 
Account  Roster. 

b Does  not  total  100 %  due  to  rounding. 


TABLE  2-2 


PARTICIPATION  IN  SURVEY  BY  ACTIVITY  AND  DIVISION 


UNIT 

PARTICIPTION 

(PERCENT) 

DIVISION: 

A 

64.5 

B 

83.3 

C 

72.7 

D 

86.5 

E 

88.9 

F 

62.5 

G 

75.0 

H 

97.3 

ACTIVITY: 

80.3 

2.2.3  Data  Collection  Procedures 


a.  The  survey  was  administered  to  the  TTS  on  29  and  30  June  1981.  The 
timir  >  *.as  selected  to  allow  maximum  exposure  to  the  system  for  familiarity, 
but  ytc  to  gather  perceptions  before  the  first  scoring  period  results  were 
available.  Since  the  first  scoring  period  ended  30  June  1981,  the  survey 
captured  perceptions  at  the  end  of  the  period,  during  which  61  projects  were 
completed  and  scored.  Makeup  sessions  were  held  on  20  and  24  July. 

b.  A  letter  signed  by  the  Assistant  Deputy  for  Technical  Operations 
was  sent  to  each  of  the  eight  division  chiefs  approximately  one  week  before 
the  scheduled  administration  date.  The  division  chiefs  were  all  aware  that 
the  survey  was  to  be  administered  at  the  end  of  the  first  scoring  year  because 
they  had  reviewed  and  approved  the  project  plan  in  the  Productivity  Measure¬ 
ment  Board  (PMB)  meeting  (12  June  1981).  At  the  TRASANA  staff  meeting  of 
division  chiefs  on  25  June,  maximum  participation  in  the  survey  was  encour¬ 
aged.  Division  secretaries  were  also  phoned  on  administration  dates  as  a 
remi nder. 

c.  The  survey  was  scheduled  to  provide  as  little  work  interruption  as 
possible,  and  to  be  as  convenient  as  possible  while  still  under  controlled 
conditions.  Four  three-hour  time  blocks  were  established  over  the  two  days 
with  two  divisions  being  scheduled  for  each  time  slot.  Analysts  could  come  to 
the  scheduled  conference  room  at  their  convenience,  complete  the  survey,  and 
leave.  The  time  schedule  was  only  to  avoid  congestion.  No  attempt  was  made 
at  schedule  enforcement,  and  all  requests  to  take  the  survey  in  a  different 
time  slot  were  honored.  The  survey  took  45  minutes  to  an  hour,  and  was  taken 
anonymously.  Group  administration  was  employed  to  control  for  discussion  of 
questions  and  answers  while  responding  and  to  achieve  a  more  representative 
sampling  than  could  be  achieved  by  a  mailing  procedure. 
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2.3  EVALUATION  OF  TPMS  FIRST  YEAR  PERFORMANCE 


The  evaluation  of  the  TPMS  first  year  performance  involved  the  following 
steps : 

°  gathering  and  analyzing  first  year  scoring  results 
°  conducting  a  literature  survey  on  productivity  evaluation 
°  developing  a  productivity  criterion  measure  (CM) 

°  col lecting  CM  data 
°  PI  evaluation  analysis 

The  methodology  employed  for  each  of  these  steps  is  discussed  separately. 


2.3.1  First  Year  Scoring  Results 


a.  TRASANA  Memo  5-4,  describes  the  TPMS  three  year  implementation, 
scoring  procedures,  and  record  keeping  requirements.  Essentially,  each  of  the 
eight  TRASANA  divisions  are  responsible  for  keeping  records  of  their  projects, 
and  for  reporting  the  results  at  the  end  of  each  scoring  year. 

b.  A  set  of  three  forms  was  developed  which  would  provide  the  neces¬ 
sary  information  from  each  division  for  this  evaluation,  as  well  as  provide  a 
convenient  summary  record  which  would  meet  the  needs  of  the  divisions.4  A 
copy  of  this  set  of  forms,  TPMS  Division  Scoring  Sheet,  TPMS  Branch  Scoring 
Sheet,  and  TPMS  Individual  Scoring  Sheet  is  in  appendix  D. 

c.  The  forms  were  distributed  to  the  division  chiefs  at  a  PMB  meeting 
with  instructions  to  furnish  a  copy  to  the  TPMS-TE  evaluation  team  not  later 
than  15  August  1981,  which  was  enough  time  for  all  scoring  records  to  be 
available. 

d.  Project  scoring  data  were  obtained  from  the  TPMS  scored  project 
log,  which  is  maintained  by  the  PMB  chairman. 

e.  After  all  scoring  data  were  received,  a  descriptive  summary  was 
prepared  and  analyzed  for  each  of  the  scoring  variables,  for  each  project,  and 
for  each  scoring  unit.  These  data  formed  the  Phase  I  TPMS-T  data  base. 


4The  final  version  forms  were  a  slighty  modified  version  of  those  developed  by 
ATAA-TE. 
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2.3.2  Literature  Survey  Summary 


A  comprehensive  literature  review  was  beyond  the  scope  of  this  study. 

Instead,  an  attempt  was  made  to  select  the  most  relevant  works  as  a  framework 
for  this  evaluation.  The  works  selected  for  review  fell  into  four  general 
categories : 

°  documents  pertaining  specifically  to  TPMS 
°  Research  and  Development  (R&D)  productivity  measurement 
°  performance  appraisal  (PA) 

°  peer  ratings 

All  available  documentation  on  TPMS  were  reviewed  for  background,  and  the  best 
available  references  on  measuring  productivity  of  R&D  or  scientist  and 
engineer  organizations  were  consulted  for  background  in  developing  the  CM. 
References  on  PA  and  peer  ratings  were  consulted  as  these  techniques  became 
important  in  developing  the  CM.  A  detailed  summary  of  the  works  consulted  is 
not  included,  because  excellent  literature  summaries  appear  elsewhere.5 
Conclusions  from  the  literature  survey  are  outlined  to  show  the  reasoning 
behind  the  specific  study  procedures  employed. 

a.  Measuring  R&D  Productivity.  The  predominant  approach  to  measuring 
productivity  among  scientists  and  engineers  is  to  opt  for  some  easily  count¬ 
able  measure  such  as  number  of  publications  (Blose  &  Stansel ,  1975;  Katzen, 
1975;  Koser,  1976;  Newburn,  1972;  Technical  Report  74-3,  USAF  AFIT,  1974;  and 
Technical  Report  78-1,  USAF  AFIT,  1978).  One  ambitious  attempt  to  compare  the 
knowledge  production  of  major  universities  failed  to  find  a  more  acceptable 
measure  than  publications  (Clark  &  Guba,  1977).  The  obvious  problem  with 
counting  products  is  that  the  quality  of  the  products  is  not  considered.  One 
high  quality  product  may  be  equal  to  several  lesser  quality  works.  Edwards 
and  McCarrey  (1973),  after  an  extensive  literature  review  concluded  that  more 
study  is  needed  on  how  to  measure  the  performance  of  research  or  knowledge 
type  workers.  The  difficulty  is  measuring  quality  or  relevant  worth.  The 
most  credible  method  of  doing  this  is  by  blind  peer  review.  This  method 
involves  presenting  a  small  group  of  widely  recognized  professionals  in  a 
specific  field  with  works  not  identified  by  author  to  be  rated  according  to 
specified  rating  criteria.  This  "blind  review"  is  the  method  used  by  many 
professional  journals  to  select  articles  for  publication  (commonly  referred  to 
as  "refereed"  journals). 


5The  works  consulted,  are  arranged  by  subject  area  in  the  bibliography.  The 
most  complete  general  reference  is  R&D  Productivity  published  by  the  Hughes 
Aircraft  Company  (1978).  Other  references  listed  in  the  bibliography 
containing  relevant  literature  summaries  include:  Christ  (1981),  Edwards  and 
McCarrey  (1973),  Schainblatt  (1981),  and  Technical  Report  79-9  (USAF  HRL, 
1979). 
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b.  Peer  Ratings.  The  most  frequent  use  of  peer  ratings  reported  in 
the  literature  is  in  training  courses.  Very  little  was  found  that  related  to 
scientists  and  engineers.  No  evidence  was  found  to  support  the  use  of  peer 
ratings  for  determining  product  quality  in  an  organization  like  TRASANA. 
Indeed,  the  problems  that  are  reported  in  attempts  to  use  peer  ratings  suggest 
questionable  validity.  It  was  anticipated  that  with  the  diverse  backgrounds 
of  the  TTS,  documented  problems  with  peer  ratings  would  be  magnified. 

c.  Performance  Appraisals.  Once  it  became  evident  that  peer  ratings 
were  not  a  viable  alternative  for  use  as  a  CM,  it  was  apparent  that  some  form 
of  supervisor  ratings  or  evaluations  would  be  necessary.  The  PA  literature 
was  searched  to  find  the  most  appropriate  factors  to  include  in  the  appraisals 
as  well  as  the  best  structure  or  format.  From  the  literature  it  was  found 
that  the  most  reliable  and  valid  appraisal  is  a  global  or  overall  rating. 

This  method  was  adopted  for  the  CM. 

2.3.3  Criterion  Measure 


a.  Official  employee  performance  appraisals  were  not  adequate  for  the 
CM  because  of  the  well  known  problems  such  as  halo  effect,  rater  bias,  and 
variation  in  rater  standards.  The  most  accurate  supervisor  judgments  were 
required,  and  these  judgments  had  to  be  quantitative  to  allow  comparison  with 
employee  PI  rankings. 

b.  The  selected  method  involved  asking  first  line  supervisors  (branch 
chiefs)  to  provide  a  confidential  rank  order  of  their  employees.  This  ranking 
was  done  in  a  private  interview.  Management  provided  the  guarantee  that  this 
information  would  remain  confidential.  Neither  superiors  nor  subordinates  had 
access  to  this  information.  To  preclude  any  chance  of  compromise,  the  confi¬ 
dential  information  from  branch  chiefs  was  destroyed  as  soon  as  the  necessary 
data  pairings  for  the  analysis  were  completed.  In  addition  to  the  ranking  of 
employees,  a  rating  was  also  obtained.  The  rating  was  an  overall  performance 
evaluation  on  a  scale  of  1  to  10,  with  5  being  average  for  TRASANA,  and  10 
being  the  most  outstanding.  These  two  confidential  data  elements  for  each 
individual,  rank  order  in  branch  and  overall  job  performance  rating,  became 
the  productivity  CM  for  the  PI  evaluation. 

2.4  TPMS  COMPONENT  FUNCTIONING 

This  portion  of  the  evaluation  effort  consisted  of  determining  how  well  the 
Productivity  Measurement  Board  (PM8),  the  Product  Review  Board  (PRB),  and  the 
TPMS  scoring  algorithm  functioned  when  considered  all  together. 

2.4.1  PMB 

Data  were  collected  from  three  sources  to  determine  how  well  the  PMB  worked 
the  first  year.  First,  a  meeting  was  observed.  Second,  the  PMB  members  were 
interviewed.  Third,  the  first  year  scoring  results  were  analyzed  to  determine 
if  the  PMB  monitoring  of  project  plans  was  achieving  its  objective  of  keeping 
manpower,  computer,  and  timeliness  estimates  from  being  inflated. 
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The  PRB  functioning  was  determined  by  interviewing  selected  PRB  members,  and 
division  chiefs,  branch  chiefs,  and  study  managers  who  experienced  PRB  scoring 
during  the  first  scoring  year.  The  scores  given  by  the  PRB  during  the  first 
scoring  period  were  also  analyzed. 

2.4.3  Algorithm 


The  TPMS  scoring  algorithm  was  analyzed  mathematically  to  clearly  identify 
factor  relationships. 


CHAPTER  3 


FINDINGS:  TECHNICAL  STAFF  SURVEY  RESULTS 


This  chapter  presents  the  results  of  the  survey  administered  to  the  TRASANA 
Technical  Staff  (TTS)  at  the  end  of  the  first  scoring  year  to  achieve  the 
first  study  objective:  to  determine  how  the  TTS  perceives  TPMS.  The 
following  study  questions  for  this  objective  are  answered: 

°  What  is  the  TTS  general  attitude  toward  TPMS? 

0  What  is  the  perceived  motivational  impact  of  TPMS? 

0  Is  TPMS  seen  as  a  viable  managerial  tool? 

Emphasis  was  on  quantifying  the  general  attitude  toward  TPMS,  determining  its 
perceived  motivational  impact,  and  assessing  the  perceptions  of  TPMS  as  a 
viable  management  tool.  Since  the  general  predisposition  of  the  TTS  would  be 
an  important  factor  in  how  a  new  productivity  measurement  system  would  be 
accepted,  organizational  climate  data  were  also  collected  in  the  survey,  and 
are  reported  in  this  chapter. 

3.1  THE  SURVEY 

a.  A  complete  copy  of  the  survey  is  in  appendix  B.  It  consists  of  91 
items  plus  a  comment  page.  The  first  14  items  pertain  to  background 
information.  The  data  obtained  from  these  items  are  summarized  in  tables  C-2 
through  C-7  of  appendix  C.  These  data  document  the  fact  that  the  TTS  is  a 
highly  educated,  experienced,  stable  work  force  consisting  primarily  of 
scientists  and  engineers. 

b.  Seventy-five  items  were  constructed  on  a  six-point  forced-choice 
Likert  scale  with  no  neutral  response  category.  The  Likert  scale  was  selected 
because  it  gives  not  only  an  agree  or  disagree  response  to  each  item,  but  also 
yields  an  intensity,  or  degree  of  agreement  or  disagreement,  which  is  much 
more  revealing  than  a  two  category  (yes  or  no)  response.  Sixty-two  items  were 
grouped  into  the  following  clusters: 

°  Attitude  toward  TRASANA 

°  Job  Satisfaction 

0  Control  of  work 

°  General  reaction  to  TPMS 

°  Fairness  of  TPMS 

0  Understanding  of  TPMS 

°  Usefulness  of  TPMS  as  a  management  tool 
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The  additional  13  items  did  not  fall  into  any  cluster,  but  addressed  specific 
points  of  interest. 

3.2  SURVEY  RESULTS 

The  results  of  the  survey  clusters  are  discussed  in  this  section.  Even  though 
cluster  scores  are  more  stable  and  contain  less  error  variance  than  individual 
item  responses,  it  is  recognized  that  review  of  the  distribution  of  responses 
to  individual  items  may  be  of  interest  to  some  readers.  Tables  C-B  and  C-9  in 
appendix  C  give  the  response  distributions  for  all  of  the  questions  in  the 
survey  except  the  demographic  items,  which  are  in  tables  C-2  through  C-7. 

3.2.1  Cluster  Means  and  Reliabilities 


a.  The  results  of  the  survey  clusters  are  in  table  3-1.  The 
reliability  coefficients  {coefficient  alpha)  range  from  0.70  to  0.86  for 
individual  clusters,  indicating  that  the  items  in  the  clusters  were  generally 
a  homogeneous  set  of  items  (all  measuring  the  same  thing).  The  reliability 
coefficient  of  0.89  for  the  entire  75  items  is  high  for  a  survey  of  this  type. 

b.  The  results  were  scored  so  that  the  higher  the  score,  the  more 
positive  the  attitude.  The  mid-point  of  the  six-point  scale  is  3.5.  Scores 
falling  in  the  3.0  to  4.0  range  are  neither  clearly  positive  nor  negative  due 


TABLE  3-1 

TTS  SURVEY  CLUSTER  { SUBSCALE)  RESULTS 


CLUSTER 

number  of 

QUESTIONS 

MEAN 

STANDARD 

DEVIATION 

sem 

ORGANIZATION: 

Attitude  toward  TRASANA 

12 

.86 

4.32 

.84 

.31 

Job  Satisfaction 

9 

.81 

4.51 

.89 

.39 

Control  of  work 

3 

.70 

4.63 

1.08 

.59 

TPMS: 

General  Reaction 

11 

.73 

3.45 

.76 

.39 

Fairness 

4 

.82 

2.96 

1.21 

.51 

Understanding 

11 

.76 

4.00 

.87 

.21 

Management  Tool 

12 

.70 

3.46 

.72 

.39 

CLUSTER  TOTAL 

62 

.88 

3.82 

.72 

.25 

SURVEY  TOTAL 

75 

.89 

3.80 

.45 

.15 
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to  possible  measurement  error. 1  The  three  clusters  measuring  organizational 
climate  are  clearly  in  the  positive  range,  showing  a  very  favorable  regard  for 
the  organization,  job  satisfaction,  and  control  of  work  (important  for  R&D 
organizations).  The  TPMS  clusters  are  less  positive.  The  general  reaction 
and  usefulness  as  a  management  tool  are  clearly  in  the  neutral  range.  The 
staff  generally  considers  itself  to  understand  TPMS,  but  does  not  think  TPMS 
is  fair. 

3.2.2  Attitude  Toward  TRASANA 

Overall,  the  staff  feels  that  TRASANA  is  a  good  place  to  work.  Ninety-four 
percent  of  the  respondents  like  working  at  TRASANA,  90  percent  are  proud  of 
TRASANA' s  products,  91  percent  think  that  TRASANA  has  high  standards,  and  87 
percent  feel  that  people  are  treated  fairly. 

3.2.3  Job  Satisfaction 

a.  In  general  the  TTS  expressed  a  high  degree  of  job  satisfaction. 
Sixty-eight  percent  indicated  they  get  a  lot  of  satisfaction  from  their  job, 
and  75  percent  report  their  job  is  challenging.  Of  those  not  experiencing 
much  personal  satisfaction  from  their  jobs,  43  percent  indicated  that  their 
skills  were  not  well  matched  to  the  requirements  of  their  jobs. 

b.  Fifty-four  percent  of  the  respondents  do  not  feel  they  get  useful 
feedback  on  job  performance.  This  is  indicative  of  a  problem  area.  Analysts 
generally  have  a  lot  of  freedom  in  how  they  do  their  job'(92  percent  say  they 
do),  and  supervisors  generally  meet  with  approval,  yet  over  half  do  not  feel 
their  supervisors  are  doing  an  effective  job  of  performance  appraisal. 
Meaningful  performance  evaluation  and  feedback  is  important  for  many  reasons, 
including  helping  persons  to  improve  themselves  through  constructive 
criticism,  and  maintaining  a  realistic  perspective  on  future  expectations, 
such  as  promotion  potential.  A  majority  (52  percent)  do  not  see  TPMS  as  being 
useful  for  individual  appraisals,  which  means  it  will  not  likely  fill  this 
void. 

3.2.4  General  Reaction  to  TPMS 

The  aggregate  response  to  this  cluster  lies  near  the  scale  midpoint--neither 
positive  nor  negative.  This  reflects  mixed  feelings  as  well  as  disagreement 
among  the  staff.  There  is  considerable  concern  that  too  much  emphasis  will 
be  put  on  TPMS  results  to  the  exclusion  of  important  job  performance  factors 
not  captured  by  TPMS  scores  (60  percent  do  not  believe  TPMS  emphasizes  the 
most  important  factors  in  productivity).  On  the  positive  side,  81  percent 

^  Any  measurement,  physical  or  psychological,  is  only  accurate  to  some 
specified  degree,  and  contains  some  tolerance  or  error.  A  band  of  confidence 
can  be  built  around  a  test  score  to  show  the  upper  and  lower  bounds  of  a 
given  score  for  a  specified  reliability  and  distribution.  This  band  of 
confidence  can  be  calculated  by  taking  a  score  plus  and  minus  the  Standard 
Error  of  Measurement  ( SEy) .  The  SEy  for  this  survey,  shown  in  the  last 
column  of  table  3-1,  reflect  that  the  scores  are  rather  stable. 
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think  TPMS  is  being  given  a  fair  chance,  69  percent  believe  that  manpower 
resources  eventually  car,  be  real  istical  ly  estimated,  61  percent  believe 
computer  resources  can  be  realistically  estimated  with  more  experience,  and  34 
percent  have  seen  some  positive  effects  of  TPMS  (basically  in  project  planning 
and  resource  allocation),  (jn  the  other  hand,  67  percent  do  not  think  it  is 
easy  to  understand,  68  percent  believe  that  it  is  not  flexible  enough  for 
TRASANA’s  variety  of  projects,  and  83  percent  believe  it  can  be  easily 
mani pul ated . 

3.2.5  Fa i rness 

The  response  on  this  cluster  was  the  least  positive.  Fifty  percent  said  TPMS 
was  not  fair  (19  percent  strongly  disagreed  with  tne  statement  that  TPMS  is 
basically  a  fair  system).  Sixty-three  percent  do  not  believe  divisions  have 
an  equal  chance,  59  percent  do  not  believe  it  is  objective,  and  61  percent  do 
not  belive  that  the  most  productive  individuals  will  get  the  best  productivity 
scores  under  TPMS. 

3.2.6  TPMS  Understanding 


Of  the  clusters  pertaining  to  TPMS,  the  response  to  this  one  was  the  most 
positive.  It  should  be  noted  that  this  cluster  contained  items  designed  to 
reveal  system  understandi ng  as  well  as  items  asking  for  staff  perceptions  of 
their  own  and  their  peer's  understanding.  The  items  designed  to  test  system 
understanding  showed  the  staff  could  be  divided  into  three  approximately  equal 
groups:  high,  medium,  and  low  understanding.  The  supervisors  had  the  highest 
level  of  understanding,  and  the  newest  employees  the  lowest.  Now  that  the 
activity  has  completed  a  scoring  cycle,  it  is  anticipated  that  understandi ng 
of  the  system  has  improved  considerably  since  the  survey  was  administered. 

3.2.7  Usefulness  as  a  Management  Tool 

Sixty-four  percent  indicated  they  thought  TPMS  would  be  a  useful  management 
tool.  There  is  general  agreement  that  management  practices  have  changed  as  a 
result  of  TPMS  (64  percent)  and  that  more  planning  is  now  being  done  (72 
percent),  but  only  33  percent  believe  management  practices  have  improved. 
Fifty-eight  percent  believe  TPMS  is  a  clear  change  in  emphasis  by  management, 
even  though  62  percent  do  not  think  it  will  motivate  people  to  be  more 
productive. 

3.3  GROUP  COMPARISONS 

3.3.1  Supervisors  Versus  Nonsupervisors 


Mean  cluster  scores  for  supervisors  and  nonsupervisors  were  compared 
stati stical ly.  In  every  case  supervisors  had  significantly  higher  (more 
positive)  scores  than  the  nonsupervisors.2  Figure  3-1  shows  the  scoring 


2 These  comparisons  were  made  with  t-tests.  Table  C-10  in  appendix  C  girts  the 
group  means  and  standard  deviation  for  each  cluster ,  as  well  as  the  c  •  -.lues, 
which  are  are  all  significant  beyond  the  .05  level. 
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profile  of  the  two  groups.  The  clusters  are  shown  along  the  bottom  or 
horizontal  axis,  and  the  cluster  mean  scores  are  scaled  along  the  vertical 
axis.  This  profile  analysis  shows  that  supervisors  have  a  consistently  more 
positive  attitude  than  nonsupervisors  on  both  the  organizational  and  TPMS 
clusters.  The  fact  that  the  two  profiles  are  almost  parallel  indicates  that 
the  supervisors  are  not  only  consistently  more  positive,  but  uniformly  more 
positive  as  well  (there  are  no  group-cluster  interaction  effects).  While  it 
is  not  surprising  that  supervisors  are  more  positive  about  the  organization, 
the  near  perfectly  parallel  profiles  indicates  that  supervisors  and 
nonsupervisors  have  the  same  relative  feelings  about  TPMS. 


CLUSTER 

MEAN 


A 


—  —  —  Supervisor 
-  Non-Supervisor 


TRASANA  JOB  CONTROL  GENERAL  FAIR-  UNDER-  TOOL 

SATIS-  OF  REACTION  NESS  STANDING 
FACTION  WORK 


CLUSTER 


Figure  3-1.  Supervisor/Non-Supervisor  Profile  Analysis 


3.3.2  Other  Group  Comparisons 


a.  Mean  cluster  scores  were  also  compared  for  persons  grouped  on  the 
following  variables: 

°  Age 

°  Time  in  Federal  Service 
0  Time  at  TRASANA 
l!  Grade 

°  Military/Civilian 
°  Division 

Most  of  these  comparisons  revealed  no  significant  differences  between  groups, 
but  a  few  interesting  trends  were  discovered. 

b.  There  was  a  tendency  for  those  who  had  been  in  the  organization 
longer  to  have  a  better  understanding  of  the  system,  to  see  the  system  as  less 
unfair,  and  to  have  a  more  positive  general  reaction  to  TPMS.  It  should  be 
recognized  that  nearly  all  of  the  supervisors  fall  in  the  grouping  or  category 
that  has  been  in  the  organization  over  nine  years.  The  same  pattern  emerged 
for  the  grade  and  time  in  federal  service  categories,  because  these  groups  are 
essentially  the  same  people. 

c.  The  military  staff  members  tended  to  be  more  positive  toward  TPMS  in 
general  than  the  civilian  staff.  They  were  also  more  inclined  to  believe  the 
system  can  work  and  to  see  it  as  less  unfair  than  their  civilian  counterparts 

d.  There  were  some  differences  among  the  divisions.  The  division  with 
the  highest  job  satisfaction  cluster  mean  (4.72)  had  the  lowest  mean  score  on 
general  reaction  to  TPMS  (3.19)  and  fairness  (2.49),  and  among  the  lowest  mean 
score  on  usefulness  as  a  management  tool  (3.27).  This  same  divisior  scored 
high  on  understanding  (4.16).  This  can  be  contrasted  with  the  division  having 
the  lowest  job  satisfaction  mean  (3.89),  which  had  much  higher  scores  for 
general  reaction  (3.63),  fairness  (3.70),  and  usefulness  as  a  management  tool 
(3.87).  This  division  did  net  have  as  high  a  mean  on  understanding  as  the 
high  satisfaction  division  (3.85  versus  4.16).  The  division  with  the  higher 
job  satisfaction  scores  seems  to  be  more  contented  with  the  status  quo  and 
less  accepting  of  TPMS  (which  is  a  change)  than  the  division  with  not  as  high 
a  level  of  job  satisfaction.  The  latter  is  more  accepting  of  TPMS  and  more 
confident  TPMS  will  be  a  positive  change,  possibly  reflecting  a  hope  of  better 
things  to  come. 
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3.4  RESPONSE  TO  FIRST  OBJECTIVE 


3.4.1  Objective  1:  To  determine  how  the  TRASANA  Technical  Staff  (TTS) 

perceives  TPMS. 

a.  Study  Question  1:  What  is  the  TTS  general  attitude  toward  TPMS? 

The  overall  reaction  to  TPMS  was  mixed.  There  was  considerable  concern  and 
uncertainty.  One  concern  was  that  some  staff  members  manipulate  the  system 
rather  successfully  and  thereby  achieve  higher  productivity  indexes  (PI)  than 
others  who  are  just  as  productive,  but  do  not  ''play  the  system"  as  well. 

There  was  rather  deep  concern  about  how  the  results  will  be  used  -  especially 
the  individual  PI  -  and  the  resulting  effect  on  morale  and  the  spirit  of 
cooperation  so  highly  valued  at  TRASANA.  Another  concern  was  that  TPMS  may 
result  in  a  long  term  decline  in  the  willingness  to  take  on  high  risk  projects 
and  to  try  to  develop  new  methodologies.  On  the  positive  side,  the  detailed 
planning  for  all  projects  was  seen  as  very  positive,  the  majority  think 
resources  can  be  realistically  estimated  with  more  experience,  and  one-third 
reported  that  they  had  seen  some  positive  effects  of  TPMS  during  its  first 
year. 


b.  Study  Question  2:  What  is  the  perceived  motivational  impact  of 

TPMS? 

Over  half  of  the  staff  do  not  think  TPMS  will  motivate  people  to  be  more 
productive,  but  they  are  evenly  divided  on  whether  TPMS  will  lead  to  a  long 
term  productivity  increase.  These  results  reflect  the  reality  that  TPMS  will 
not  have  the  same  impact  on  all  people.  What  the  total  impact  will  be  depends 
to  some  extent  on  how  the  system  is  changed  to  maximize  the  positive  and 
minimize  the  negative  factors. 

c.  Study  Question  3:  Is  TPMS  seen  as  a  viable  managerial  tool? 

Yes.  This  was  seen  as  the  most  positive  aspect  of  TPMS.  In  particular,  the 
detailed  planning  required  for  all  projects  in  the  form  of  the  project  plan 
was  seen  as  very  useful . 

3.4.2  Organizational  Climate 

The  TRASANA  staff  feel  very  positive  about  the  organization,  see  their  work  as 
challenging,  and  like  the  high  standards  of  TRASANA.  There  is  a  concern  felt 
by  both  supervisors  (division  and  branch)  and  staff  that  TPMS  may  have  a 
detrimental  effect  on  the  spirit  of  cooperation  and  on  the  willingness  of  the 
staff  to  take  on  the  very  difficult  and  risky  projects  that  in  the  past  have 
been  welcomed  as  a  challenge. 
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CHAPTER  4 


FINDINGS:  FIRST  YEAR  RESULTS 


This  chapter  presents  the  data  collected  to  achieve  the  second  study 
objective:  to  determine  the  degree  to  which  TPMS  is  a  meaningful  measure  of 
productivity.  The  following  study  questions  for  this  objective  are  answered: 

°  To  what  extent  do  TPMS  ratings  agree  with  confidential  supervisory 
productivity  assessments  for  individuals  and  branches? 

°  Does  TPMS  have  generally  the  same  impact  on  the  different  TRASANA 
organizational  divisions? 

°  What  a^e  the  functional  relationships  among  the  scoring  algorithm 
variables? 

°  What  is  the  operational  impact  of  the  Productivity  Measurement  Board 
( PMB)  and  Product  Review  Board  (PRB)? 

The  first  step  in  this  effort  was  to  capture  all  of  the  first  year  scoring 
results.  These  data  are  presented  for  the  sixty-one  scored  projects,  as  well 
as  by  TPMS  scoring  factor.  The  productivity  index  (PI)  for  individuals  and 
organizational  units  (branches,  divisions)  is  examined,  the  individual  PI 
evaluation  analysis  is  presented,  and  scoring  problems  revealed  from  analysis 
of  the  first  year  scoring  data  are  discussed.  The  analysis  of  the  TPMS 
functioning  components  addresses  the  Product  Review  Board  and  Productivity 
Measurement  Board.  The  results  of  the  mathematical  analysis  of  the  TPMS 
scoring  algorithm  are  in  this  chapter.  Since  the  mathematical  analysis  is 
somewhat  technical  and  requires  basic  familiarity  with  mathematics  and  error 
analysis  techniques,  it  is  in  appendix  E. 

4.1  FIRST  YEAR  SCORING  RESULTS 

4.1.1  Scoring  Results  by  Project 

Table  4-1  shows  the  project  scores  (PS)  for  the  61  projects  which  were 
completed  and  scored  between  1  October  1980  and  30  June  1981.  The  highest  was 
2.376  and  the  lowest  was  1.046.  The  mean  project  score  was  1.681  and  the 
standard  deviation  was  0.258.  The  maximum  possible  PS  is  2.614,  and  the 
minimum  is  0.504.  The  baseline  score  for  a  project  on  time,  on  resources,  and 
of  average  quality  (defined  as  1.3)  is  1.56.  The  score  assigned  to  the 
projects  on  each  scoring  variable  and  the  productivity  points  (PP)  earned  also 
are  shown  in  the  table. 

4.1.2  Manpower  Scoring  Results 

The  manpower  utilization  factor  (R)  is  scaled  as  a  piecewise  linear  increasing 
function  of  the  ratio  of  planned  (Rp)  to  actual  (R/\)  manpower  for  the  project. 
It  has  0.7  as  the  lower  limit  and  1.2  as  the  upper  limit  value.  A  detailed 
discussion  of  the  scoring  algorithm  is  in  appendix  E. 
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NOTE:  Not  all  scores  are  shown  to  the  same  number  of  significant  digits 
because  there  was  variation  in  the  officially  recorded  scores. 


KEY:  PS-Project  Score 

Rp-Planned  Manpower 
Rfi-Actual  Manpower 


R-Manpower  Utilisation 
C-Computer  Utilization 
T-Timeliness 


Q-Quality 

PP-  Productivity  Points 


a.  The  R  factor  results  for  60  projects  are  shown  in  histogram  form  in 
figure  4-1. ^  The  planned  manpower  ranged  from  1.2  to  125  technical  man  months 
( TMM)  with  an  average  of  19.8.  The  actual  manpower  utilized  varies  from  0.1 
to  115.11  TMM  with  an  average  of  17.9.  Values  for  R  ranged  from  0.861  to  the 
maximum  allowable  score  of  1.2.  The  average  was  1.063  and  the  standard 
deviation  was  0.093.  Twenty-five  percent  of  the  projects  received  an  R  score 
of  approximately  1.0,  which  means  the  R^  was  equal  to  the  Rp.  Two  thirds  of 
the  projects  had  an  R  value  greater  than  1.0,  which  means  they  received  a 


* As  shown  in  table  4-1 ,  one  project  did  not  have  scores  on  R,  T,  Q,  or  C. 
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"bonus"  for  R/\  being  less  than  Rp.  Thirteen  percent  of  the  projects  received 
the  maximum  R  score  of  1.2.  To  achieve  the  maximum  R  value,  R/\  must  be  less 
than  or  equal  to  one  half  of  Rp.  For  the  eight  projects  achieving  the  maximum 
R  score,  Rp  varies  from  2.033  to  12.0  times  R^. 

b.  If  manpower  estimates  were  reasonably  accurate,  the  R  distribution 
would  be  symmetrical  around  a  mean  of  approximately  1.0,  or  most  projects 
would  have  R/\  approximately  equal  to  Rp,  with  as  many  projects  falling  under 
as  over  the  estimates.  The  probability  that  43  of  60  projects  (72  percent) 
would  be  under  the  estimated  manpower  requirement,  assuming  accurate 
estimations,  is  approximately  0.0004,  or  4  times  in  10,000.  It  can  only  be 
concluded  that  Rp  was  consistently  overestimated. 

c.  There  were  several  factors  driving  the  consistent  tendency  to 
estimate  required  manpower  high.  Such  accurate  estimating  as  required  by  TPMS 
was  new  to  the  staff,  and  it  takes  some  experience  to  get  good  at  such  a 
difficult  task.  Even  with  practice,  estimates  for  complex,  risky  projects  can 
never  be  completely  accurate.  When  this  is  considered  along  with  the  fact 
that  TPMS  rewards  overestimating  with  higher  project  scores,  and  therefore 
tends  to  punish  either  accurate  or  underestimating  with  lower  scores  (less 
reward),  the  driving  force  for  overestimating  becomes  clear.  The  only 
explicit  punishment  in  this  function  is  for  substantial ly  uncerestimating 
manpower  requirements  which  results  in  earning  points  below  the  base  level  of 
1.0. 


d.  There  are  two  system  checks  on  Rp.  Before  a  project  plan  can  be 
entered  into  TPMS  for  scoring,  the  responsible  division  chief  must  show  that 
his  total  manpower  committed  for  all  approved  projects  is  not  more  than  125 
percent  of  available  manpower  (125  percent  rule).  The  other  check  is  that  the 
project  plan  must  be  reviewed  and  approved  by  the  PMB.  The  purpose  of  this 
review  is  to  check  for  realistic  and  accurate  resource  estimates  and 
schedules.  These  two  checks  were  not  effective  in  controlling  the  manpower 
estimates. 

4.1.3  Product  Timeliness  Scoring  Results 

When  a  project  plan  is  submitted  to  the  PMB  for  approval,  it  must  contain  a 
proposed  schedule  for  scoring.  When  the  project  is  completed  (including 
either  a  0  score  of  1.3  or  higher  or  PRB  approval  for  release),  a  score  is 
assigned  based  on  the  relationship  of  the  actual  completion  date  to  the 
planned  completion  date.  The  values  for  this  variable  are  0.8  to  1.1,  with 
0.8  being  the  lowest  score  for  being  late  and  1.1  being  the  highest  possible 
score  for  completion  ahead  of  schedule. 

a.  The  distribution  of  the  timeliness  (T)  scores  is  shown  in  figure 
4-2.  Thirty-two  of  the  60  projects  (53  percent)  received  a  score  of  1.0, 
which  means  they  were  delivered  exactly  on  schedule.  Six  (10  percent) 
received  the  minimum  score  of  0.8  and  11  (18  percent)  received  the  maximum 
value  of  1.1.  The  mean  T  was  0.997  with  a  standard  deviation  of  0.083.  Ten 
(17  percent)  were  late,  and  18  (30  percent)  were  delivered  ahead  of  schedule. 
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b.  One  reason  for  the  large  number  of  projects  beino  deliverer!  on 
schedule  is  external  deadlines.  In  many  cases  the  reports  are  of  lirited 
value  if  not  available  to  decision  makers  by  a  specified  date,  even  if  the 
scope  of  the  analysis  must  be  reduced.  In  such  cases,  the  project  scope  must 
he  tailored  to  the  deadline  and  the  report  published  on  tine. 

4.1.4  Product  Quality  Scoring  Results 

When  a  project,  is  completed,  the  Product  Review  Roard  (PRR)  convenes  to  review 
the  product  generated  hv  the  project.  This  product  is  typically  a  report 
which  is  distributed  to  the  board  members  in  advance.  The  three  permanent 
board  members  assign  a  product  quality  score  (0)  that  can  range  from  1.0  to 
1.5.  A  minimum  score  of  1.3  is  required  for  the  report  to  be  released  for 
distribution  outside  the  activity.  Reports  achievino  a  0  score  of  less  than 
1.3  require  revision  before  release.  The  0  score  for  the  product  does  not 
change  as  a  result  of  PRR  required  revisions. 

a.  The  hist.oqram  in  figure  4-3  shows  the  distribution  of  the  50 
quality  scores  for  the  first  year.  The  distribution  is  negatively  skewed  with 
a  mean  of  1.300  and  a  standard  deviation  of  O.Ofid.  The  scores  ranged  from 
1.15  to  1.5.  Seventeen  scores  (28  nercent)  were  below  the  established  minimum 
score  for  release  (1.3).  This  means  that  the  PRR  required  substantial 
revisions  or  "polishing"  on  over  one-fourth  of  the  products.  This  quality 
control  function  of  the  PRR  thus  had  a  major  impact  during  this  nine  month 
peri od. 


h.  The  PRR  was  not  new  with  TPMS,  but  assigning  numerical  scores  was. 
It  has  long  been  recognized  as  a  need  because  of  the  highly  technical  nature 
of  the  analysis  performed  by  TRASANA  which  must  typically  be  documented  in  a 
renort  understandable  by  readers  who  often  do  not  have  the  training  necessary 
to  interpret  a  very  technical  report.  This  requirement  to  present  a  large 
amount  of  data  and  complex  analyses  in  relatively  non-technical  terms  accounts 
for  many  of  the  scores  below  1.3. ? 

c.  The  quality  score  is  the  only  variable  in  the  TPMS  scoring  function 
which  is  totally  independent  of  the  project  team.  While  the  team  is  present 
during  the  review  and  actively  interacts  with  the  board,  the  board  decides  the 
score.  The  fact  that  0  is  independent  of  the  team  would  seem  to  make  this 
factor  a  candidate  for  heavier  weighting  in  the  PS;  however,  there  are  three 
formidable  problems  with  0:  (1)  adverse  morale  impact,  (?)  historical 

difficulty  in  achieving  high  reliability  for  such  scoripg  tasks,  (3)  the 
inability  of  a  few  individuals  (any  individuals)  to  be  thoroughly  knowledge¬ 
able  of  all  the  relevant  issues  and  problems  involved  in  conducting  the  larqe 
number  of  diverse  projects  typical  for  a  year  at  TRASANA. 


vThf>  perpon  who  perved  an  PRR  hoa*d  ehairrerpon  during  de  first,  uear  wap 
i ntervie>,Jed  upon  hie  departure  from  TRASANA  phnrtlu  after  de  first  peorina 
period  ended.  Tt  wap  bits  eondupinn  that  dip  wap  the  rape. 
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(1)  The  most  serious  of  the  problems  with  0  is  its  adverse  inpact 
on  norale.  This  problem  was  consistently  evident  from  division  chiefs  to 
lower  grade  team  participants.  In  confidential  interviews  with  division 
chiefs,  branch  chiefs,  and  study  nanaqers,  the  comments  about  the  PRB  and  0 
score  were  like  the  sane  recordinn  replayed.  The  criticisms  were  that  the 
board  in  qeneral  does  not  have  the  hroad  ranqe  of  technical  expertise  and 
historical  knowledqe  of  each  project  to  comprehensively  evaluate  the  project 
effort.  Furthermore,  there  is  no  consistent  PRB  assessment  and  consideration 
of  innovation,  creativity,  risk,  and  potential  impact  of  each  product.  The 
bottom  line  is  that  the  TTS  perceives  that  products  are  scored  based  on 
format,  composition,  and  editorial  considerations ,  not  quality  of  analytical 
effort. 


(?)  There  was  no  way  for  the  evaluation  team  to  objectively 
determine  the  exact  process  used  to  arrive  at  a  n  score  for  a  product; 
however,  available  research  on  qradinq  in  neneral  clearly  documents  the  fact 
that  there  is  considerable  error  in  makinq  fine  discriminations  of  the  type 
required  of  the  PRR.-7  While  the  maqnitude  of  error  can  be  reduced  by  the  use 
of  detailed  qradinq  criteria  or  standards,  it  can  not  be  eliminated.  The 
absence  of  such  standards  in  scorinq  by  the  PRB  lends  credence  to  the  qeneral 
contention  that  there  is  arbitrariness  in  scorinq. 

(3)  The  basic  issue  reduces  to  whether  the  three  permanent  votinq 
PRB  members  auqmented  by  ad  hoe  members  for  specific  projects  can  consistently 
apply  the  same  exact  standards  with  the  same  level  of  objectivity  and  accuracy 
to  each  and  every  product.  If  this  can  in  fact  be  done,  the  D  score  accu¬ 
rately  reflects  the  relative  product  quality.  If  not,  adjustments  must  be 
made  in  the  0  scorinq  procedure.  The  fact  that  there  is  very  little  confi¬ 
dence  in  the  relative  0  rankinq  of  products  by  those  who  scored  hiqh  as  well 


''Tn  aeneral,  it  has  been  shown  that  professionals  who  grade  or  snore  works, 
and  then  repeat  the  task  with  the.  same  products  after  a  substantial  time 
lapse  do  not  show  a  high  level  of  agreement  with  themselves .  Furthermore, 
papers  snored  by  different  professionals  do  not  reneive  the  same  grades . 
^actors  such  as  knowledge  of  author  also  affects  grades,  regard! ess  of 
Quality.  A  discussion  of  these  and  other  related  aradina  nroblems  nan  be 
found  in  many  educational  psychology  hooks  such  as  Gage  and  Berliner  (1975), 
or  introductory  measurement  and  evaluation  hooks  such  as  Wehrens  and  Lehmann 
(1977). 

^TRASANA  memos  5-4  (Den  80)  and  5-5  flay  81)  do  in  fact  provide  policy 
guidance  and.  standards  for  product  scoring.  iJnfortunatel  u,  reauiring  the  PRB 
to  determine  if  appropriate  methodoloau  was  employed  does  not  provide  any 
help  in  making  that  determination.  Th.e  predominance  of  scientific  evidence 
suggests  that  the  PRB  (or  any  such  body)  is  not  capable  of  making  such  fine 
distinctions  as  the  difference  betioeen  a  1.72,  a  1.73,  and  a  1.34  report  with, 
a  high  level  of  consistency .  This  does  not  reflect  negatively  on  the  PRB 
members,  hut  merely  points  out  that  they  are  attemptiyig  to  do  a  nearly 
impossible  task.. 
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as  low,  and  by  nearly  all  of  the  supervisors  in  the  orqan i zat i on ,  makes  a 
strong  case  for  revised  procedures,  even  if  only  for  employee  morale.  This  is 
even  more  evident  when  it  is  realized  that  the  only  project  to  score  1.5  was 
not  an  analytical  effort,  and  produced  neither  a  report  nor  a  briefing  to  be 
scored.5 

4.1.5  Computer  Efficiency  Scoring  Results 

The  computer  efficiency  factor  (C)  score  is  computed  in  a  manner  similar  to 
the  R  score.  The  values  for  C  range  from  0.9  for  underestimating  computer 
requirements  to  1.1  as  the  maximum  efficiency  score.  Of  the  60  projects 
receiving  a  C  factor  score,  51  scored  1.0,  two  were  below  1.0  and  seven  were 
above  1.6.  One  project  received  the  minimum  score  of  0.9,  and  four  the 
maximum  score  of  1.1.  This  factor  was  relatively  unimportant  in  the  scoring 
because  the  unit  of  measure  changed  during  the  scoring  period  from  central 
processing  unit  (CPU)  to  standard  unit  of  processing  (SUP).  Since  most 
project  plans  estimated  computer  requirements  in  CPU's,  and  there  is  no  direct 
CPU-SUP  conversion,  the  majority  of  projects  were  assigned  the  C  score  of  1.0 
administratively. 

4.1.6  Priority  Factor  Scoring  Results 

The  priority  factor  (P)  was  included  in  the  scoring  formula  as  an  incentive  to 
strive  for  products  meeting  the  needs  of  customers  external  to  TRASANA.  A 
value  of  1.2  is  assigned  to  all  projects  conducted  in  support  of  external 
customers,  and  all  internal  projects  are  assigned  a  priority  that  can  vary 
from  1.0  to  1.2.  This  factor  did  not  work  well.  The  PMB  was  unable  to  find 
an  equitable  way  to  assign  priorities.  The  P  score  thus  became  a  constant 
value  equal  to  1.2  for  ail  projects. 

4.1.7  Project  Score  Results 

a.  The  distribution  of  the  61  project  scores  for  the  first  year  of 
TPMS  is  shown  in  figure  4-4.  The  minimum  possible  PS  is  0.504  and  the  maximum 
is  2.614.  The  negatively  skewed  distribution  shows  that  projects  tended  to 
exceed  the  base  score  value  of  1.56  (1.3  Q  score  and  on  planned  resources  and 
schedule).  The  mean  score  was  1.681  and  the  standard  deviation  was  0.258. 

One  project  received  the  base  score,  and  42  (69  percent)  projects  scored  above 
the  base  score. 

b.  Table  4-2  shows  the  relative  effect  on  the  PS  of  the  same  unit 
change  (0.1)  in  the  Q,  T,  R,  and  C  factors.  The  effect  of  the  increase  is 
less  for  the  Q  factor  than  for  the  other  factors.  This  is  because  the  Q  scale 
has  a  greater  range  than  the  other  factors,  so  a  0.1  change  has  less  impact 
than  it  does  on  the  more  compressed  range  values.  The  proper  strategy  for 
maximizing  PS  is  not  to  concentrate  on  Q,  but  to  concentrate  on  T.  R,  and  C. 


5This  project  was  monitoring  of  the  installation  of  smoke  detectors  in 
TRASANA .  The  next  highest  score  was  1.42. 
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4.1.8  Scoring  Results  by  Pivision 

Table  4-3  shows  the  mean  and  standard  deviation  for  R,  C,  T,  0,  and  PS  b.v 
TRASANA  nivision.  The  number  of  projects  listed  for  each  division  includes 
scored  projects  for  which  the  division  was  primarily  responsible.  For  R,  C, 
and  T,  the  closer  the  mean  is  to  1.0,  the  more  accuracy  there  was  in  the 
estimates.  Futhermore,  the  more  inaccurate  they  are  on  the  positive  side 
(overestimates),  the  higher  the  division  PI.  The  smaller  the  standard 
deviation,  the  less  spread  there  was  in  the  scores  for  that  factor.  The 
average  project  scores  are  not  the  same  as  the  division  Pi's.  The  PI  is  a 
weighted  average  and  also  includes  overhead. 

4.?  PRODUCTIVITY  INDEX 

The  PS  is  designed  to  reflect  t.!  e  efficiency  and  effectiveness  of  a  team  of 
analysts  in  completing  a  specific  project.  Another  way  of  stating  this  is 
that  the  PS  is  a  productivity  index  (PI)  for  one  project.  The  PS  can  be 
thought  of  as  a  team  PI.  TPMS  defines  PI  as  a  weighted  averaqe  of  project 
scores.  This  weighted  averaqe  is  calculated  for  individuals  (the  average  of 
productivity  from  different  projects)  and  for  organi zational  units.  In  the 
following  discussion,  the  Pi's  calculated  for  the  first  year  are  summarized, 
and  then  the  PI  evaluation  analysis  is  presented. 


TABLF  4-3 

AVFRAfiF  PROJECT  AND  FACTOP  SCORE  RY  H 1 7 1  SI  ON 


I  DIVISION 

PROJECTS 

R 

r. 

T 

- 0 - 

PS  i 

i  a 

6 

1.019 

1 

.983 

1.290 

1 

1.571  1 

(.093) 

(o) 

(.098) 

(.091) 

(.353)  | 

| 

B 

10 

1.078 

l 

1.029 

1.303 

1.736  j 

(.062) 

(0) 

(.046) 

(.068) 

(.185)  1 

!  c 

10 

1.078 

1.029 

1.009 

1.292 

1.742  1 

(.088) 

(.044) 

(.101) 

(.044) 

(.290)  j 

1  n 

8 

1.049 

.986 

.995 

1.287 

1.  583  1 

(.125) 

(.035) 

(.095) 

(.056) 

(.210) 

F 

12 

1.090 

1.008 

.975- 

1.353 

1.744 

(.103) 

(.029) 

(.097) 

(.066) 

(.294) 

F 

2 

1.090 

1 

.950 

1.30 

1.615 

(.156) 

(0) 

(.071) 

(.071) 

(.261) 

0 

3 

1.071 

1.050 

1.035 

1.25 

1.696 

(.072) 

(.071) 

(.049) 

(.071) 

(.153) 

1 

H 

9 

1.015 

1.008 

.99 

1.317 

1.629 

(.077) 

(.023) 

(.078) 

(.085) 

(.260) 

TOTAL  n 

*0 

1.063' 

1.008 

.997 

1.309 

1.681 

(.093) 

(.031) 

(.083) 

(.069) 

(.259) 

WOTR;  ^h.e  top  number*  in  the  table  is  the  mean  value,  and  the  number  in 

parentheses  is  the  standard  deviation. 

4.2.1  Individual  PI  Results 

a.  Fiqure  4-5  shows  the  distribution  of  individual  Pi's  for  the  first 
scorinq  period.  Only  Pi's  includinq  at  least  one  man-month  are  included. 

This  was  necessary  to  eliminate  spuriously  hiqh  Pi's  that  were  primarily  an 
artifact  of  system  start,  up.  For  example,  one  individual  received  O.ll 
productivity  points  on  0.07  TMM,  and  also  collected  2.0  discretionary  points 
(DP).  The  resultinq  PI  of  30.14  was  unrealistically  inflated  because  the 
individual  only  charqed  a  fraction  of  a  month  to  scored  projects.  Inflated 
and  unrepresentati ve  Pi's  emerqed  in  several  instances  because  projects 
initiated  under  TPMS  were  not  completed  by  the  end  of  the  first  scorinq 
period.  This  situation  should  occur  less  frequently  in  future  years,  since 
all  projects  will  he  covered  by  TPMS,  and  therefore  the  number  of  people  not 
workinq  on  scored  projects  in  any  year  should  decline.  There  were  15?  persons 
with  Pi's  meetinq  the  one  TMM  charqed  to  scored  projects  criterion.  The  mean 
of  the  distribution  was  1.689  and  the  standard  deviation  was  f).?7?.  The 
values  ranqed  from  1.09  to  3.49. 


b.  A  better  understanding  of  the  individual  PI  can  be  achieved  by 
examining  various  groupings  of  individuals.  Table  4-4  shows  scoring 
information  for  the  nine  individuals  with  Pi's  exceeding  2.0.  The  project 
column  in  the  table  shows  the  number  of  scored  projects  to  which  the 
individual  charged  time,  and  TMM  is  total  time  charged  against  those  projects. 
Two  of  the  individuals  achieved  the  high  PI  erroneously,  illustrating  the 
difficulty  in  monitoring  the  tedious  and  numerous  numerical  operations 
required  by  the  system.  Such  errors  were  not  atypical  of  the  record  keeping 
and  computations. 

c.  Pi's  for  the  top  nine  scorers  were  computed  without  judgmental  and 
discretionary  awards,  and  are  shown  in  the  last  column  of  table  4-4.  These 
figures  show  that  none  of  the  nine  would  have  achieved  Pi's  over  2.0  without 
the  additional  bonuses  and  several  approach  the  mean  PI  of  1.689.  Since  the 
bonus  awards  are  at  the  discretion  of  individual  supervisors  who  make  the 
awards  without  specified  performance  standards,  it  is  not  clear  what 
significance  should  be  attributed  to  being  a  top  scorer. 

d.  The  top  nine  individual  Pi’s  were  contrasted  with  the  top  ten  - 
scoring  projects.  Table  4-5  shows  the  results.  The  rows  of  the  table  refer 
to  individuals  (the  top  nine)  and  the  columns  to  projects  (the  top  ten).  The 
entry  in  the  table  gives  the  percent  of  the  individual's  TMM  which  was  charged 
to  each  specific  project  and  the  discretionary  points  which  were  awarded  on 
each  project.  For  example,  the  table  shows  that  the  person  with  the  seventh 
highest  PI  (2.049)  charged  85  percent  of  his  TMM  to  the  tenth  highest  scoring 
project.  The  interesting  thing  about  the  table  is  the  large  number  of  blank 
squares.  For  the  most  part,  the  highest  scoring  individuals  did  not  work  on 
the  highest  scoring  projects. 

e.  In  constrast  to  the  top  nine  scoring  individuals  the  nine  lowest 
scoring  individuals  were  also  examined.  Table  4-6  shows  the  scoring  of  these 
individuals.  The  three  lowest  scores  were  achieved  by  individuals  who  had 
left  the  activity  by  the  end  of  the  scoring  period.  Each  of  the  three 
received  negative  judgmental  awards.  For  all  but  one  project,  the  lowest 
possible  award  was  given,  which  was  30  percent  less  than  the  charged  manpower 
award.  Of  the  six  still  at  TRASANA,  the  grade  distribution  is  noteworthy: 
one  GS-12,  two  GS-ll's,  and  three  GS-9's.  The  three  GS-9's  were  interns 
(trainees).  Recall  that  of  the  seven  civilians  in  the  top  nine  scorers,  there 
was  one  GS-12  and  six  GS-13's.  The  relationship  of  grade  and  PI  is  examined 
more  thoroughly  below.  To  conclude  the  discussion  of  the  bottom  nine  it 
should  be  noted  that  individual  7  achieved  the  low  PI  in  an  unusual  way. 

After  the  productivity  points  had  been  computed  and  distributed,  this  intern 
continued  to  charge  time  to  the  project.  This  resulted  in  the  PI  decreasing 
from  a  rather  respectable  1.65  to  the  1.34  shown  in  the  table. 

f.  In  theory  TPMS  makes  no  distinction  between  military  and  civilian, 
the  grade  of  the  individual,  or  the  division  to  which  the  individual  belongs. 
The  first  year  scoring  data  were  analyzed  to  determine  if  there  were  in  fact 
any  such  relationships. 
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TABLE  4-4.  SCORING  OATA  FOR  THE  NINE  HIGHEST  INDIVIDUAL  PI'S 


eMl  judgmental  awards  to  theee  individuate  were  positive ,  which  means  they  received  more  PP'e  than  the  award  for 
their  charged  manpower.  For  this  to  occur »  the  equal  number  of  PP'e  were  withheld  from  other  pereone  via  negative 
judgment  awards. 


individuals  whose  PI  contained  inappropriate  discretionary  productivity 
:  computational  error • 


TABLE  4-6.  SCORING  DATA  FOR  THE  NINE  LOWEST  INDIVIDUAL  PI 


1 


^All  judgmental  awards  to  these  individuals  were  negative ,  and  in  4  of  the  5  cases  they  received  the  lowest  award 
they  before  could  be  given,  which  is  30%  below  the  award  for  their  charged  manpower.  All  three  departed  the 
activity  before  scores  were  computed. 


(1)  Table  4-7  shows  the  conoarison  data  for  military  and  civilian 
Pi's.  There  was  no  difference  in  the  treatment  of  military  and  civilians  by 
TPMS. 


TABLE  4-7.  COMPARISON  OF  MILITARY  AND  CIVILIAN  PI 


CATFOORY 

N 

MEAN 

|  STANDARO 

MINIMUM 

MAXIMUM  1 
_  1 

1 

PI 

!  HF  V I  AT  ION 

! 

!  MILITARY 

?9 

1.60 

1 

i  0.?? 

I 

1.09? 

?.  09fi 

1 

1 

i 

I 

CIVILIAN 

! 

1?3 

1.70 

1 

1  0.?8 

1 

1 

1.094 

3. 40 

1 

I 

! 

1 

(?)  The  PI  summary  data  for  three  civilian  grade  categories  are  in 
table  4-8.  These  data  show  a  relationship  between  grade  level  and  individual 
PI.  There  was  a  statistically  significant  difference  between  the  OS-7  to  11 
group  and  the  OS-13  group.  7  Either  the  PI  is  measuring  a  real  difference  in 
productivity  by  grade  level,  or  there  is  a  difference  in  the  treatment  of 
grade  levels  in  the  application  of  TPMS. 


TABLE  4-8.  COMPARISON  OF  PI  RY  CIVILIAN  ORADE 


OS  LEVEL 

N 

MEAN  ' ' 
PI 

STANDARD 

DEVIATION 

MINIMUM 

MAXIMUM 

7  -  11 

?5 

1.57 

.?? 

1.09 

1.98 

1? 

43 

1.08 

.19 

1.39 

?..  ?4 

13 

49 

_ 

1.77 

_ 

.39 

1.40 

3.49 

P’Vk.in  comparison  Was  mads  hit  t-test ;  t  =  0.#?.,  p  =  .336. 


'The  statistical  test  applied  Was  Analysis  of  Variance  (A NOVA  ) ;  f  =  4. P.37, 
p  <  .Op.  The  multiple  nanpe  test  applied  was  Scheffe  at  the  .05  level. 


(3)  Table  4-9  qives  the  individual  PI  comparisons  by  TRASANA 
division.  GS-14's  and  1 5 ' s  are  not  included  because  their  Pi's  are  primarily 
branch  or  division  Pi's.  Statistical  analysis  revealed  no  significant 
differences  among  division  mean  Pi's.5  There  was  no  advantage  as  far  as 
individual  PI  goes  to  being  in  any  one  of  the  eight  divisions. 


TABLE  4-9.  COMPARISON  OF  PI  BY  DIVISION 
(Civilian  -  GS-13  or  Lower) 


DIVISION 

N 

MEAN 

PI 

STANDARD 

DEVIATION 

MINIMUM 

MAXIMUM 

A 

19 

1.65 

0.25 

1.09 

2.10 

B 

17 

1.70 

0.11 

1.56 

1.97 

C 

18 

1.71 

0.21 

1.40 

2.24 

D 

13 

1.68 

0.12 

1.54 

1.85 

E 

13 

1.84 

0.62 

1.40 

3.49 

F 

7 

1.64 

0.39 

1.40 

2.42 

G 

12 

1.71 

0.23 

1.38 

1.87 

H 

19 

1.63 

0.24 

1.29 

2.05 

TOTAL 

118 

1.69 

0.29 

1.09 

3.49 

4.2.2  Individual  PI  Evaluation  Analysis 

The  extent  to  which  the  individual  PI  reflects  productivity  can  only  be 
determined  by  comparing  the  PI  results  with  an  independent  productivity 
measure  or  criterion  measure  (CM).  The  results  of  the  PI  evaluation  analysis 
are  presented  next.  Since  CM  data  are  sensitive  and  were  collected  under  the 
agreement  with  branch  chiefs  that  the  data  would  remain  confidential,  the  data 
presented  are  summarized  sufficiently  to  avoid  the  possibility  of  individual 
or  branch  identification. 


8 This  comparison  was  by  AfJOVA,  F  =  0.749,  P  =  0.63. 
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a.  Table  4-10  shows  the  number  of  people  utilized  in  the  PI  evaluation 
analysis  by  division.  The  125  persons  who  had  both  a  useable  PI  (TMM  >  1.0) 
and  CM  score  was  83  percent  of  the  total  number  with  a  useable  PI,  which 
excludes  division  chiefs,  branch  chiefs,  deputies,  and  technical  advisors.5 


TABLE  4-10.  PI  EVALUATION  PARTICIPATION  BY  DIVISION 


DIVISION 

PERSONNEL 
AVAILABLE  a 
FOR  ANALYSIS 

NUMBER  WITH 
USEABLE  PP 

NUMBER  WITH 
USEABLE  PI  AND 
CM  (%) 

A 

26 

25 

20  (80) 

B 

20 

20 

18  (90) 

C 

27 

20 

20  (100) 

\P 

34 

21 

13  (62) 

E 

32 

21 

17  (81) 

F* 

24 

7 

7  (100) 

G 

17 

14 

10  (71) 

H 

31 

23 

20  (87) 

OVERALL 

211 

151 

125  (83) 

aTTS  excluding  division  chiefs ,  branch  chiefs,  deputies, 
technical  advisors,  and  identified  new  arrivals. 

b More  than  1.0  TMM  charged  to  scored  projects. 

°One  branch  had  no  branch  chief. 

d-There  were  only  two  scored  projects  in  this  division . 


qThe  PI  sample  was  representative  of  the  TTS.  The  representativeness  of  the 
PI  sample  by  grade  was  tested  by  Chi-Square.  X 5  =  4.08,  df  =  7,  which  is  not 
statistically  significant  at  the  p  <  .05  level  (X%  =  14.07).  Table  C-ll  in 

7  )  05 

appendix  C  shows  the  composition  of  the  PI  sample  and  TTS  by  gvade . 


h.  CM  data  wppp  collected  fron  nineteen  hranch  chiefs.  The  CM 
consisted  of  two  separate  data  elements,  first,  the  branch  chief  rank  ordered 
the  individuals  in  his/her  branch  on  the  basis  of  overall  iob  performance. 
Next,  each  person  in  the  branch  was  assioned  a  numerical  ration  on  a  scale  of 
1  to  Id.  The  scale  was  defined  as  10  beinq  a  "superstar,"  and  one  be' in  the 
opposite  of  10.  Five  was  defined  as  the  averaoe  for  TRASAOA.  The  rank  order 
was  obtained  for  commit  inn  a  rank  order  correlation  with  the  f*l  rank  order. 

The  ratine  was  obtained  to  determine  the  level  of  aqre°ment  of  branch  chiefs 
in  ration  standards. 

c .  Table  4-11  shows  the  nercentane  of  analysts  rated  in  each  nf  the  1 

to  10  catenories  hv  the  branch  chiefs.  The  total  number  of  ratinqs  in  the 
tahle  is  160,  which  includes  those  individuals  rated  by  branch  chiefs  who  did 
not  enter  the  analysis  due  to  the  lac>  of  a  useable  PI.  In  all  but  three 
branches,  CM  measures  were  obtained  before  °I's  were  computed.  The  mean 
ratinq  by  branch  ranoed  from  a  hiqh  of  to  a  low  nf  6,?5,  with  a  mean  of 

7.07,  a  standard  deviation  of  0.70  and  a  median  of  7.0.  These  data  show  that 
the  ten  point  confidential  ratinq  achieved  the  desired  discrimination  of 
performance  levels. 

d.  A  rank  order  correlation  was  computed  between  the  CM  rank  order  and 
the  3I  rank  order  for  nineteen  branches.  Table  4-1?  shows  the  resultinq 
correlation  coefficients  from  hiqhest  to  lowest.  The  N  is  the  number  of 
individuals  in  each  branch  who  had  both  of  the  necessary  scores  for  the 
correlation.  The  third  column  in  the  table  shows  the  necessary  absolute  value 
for  the  N  for  statistical  significance  at  the  .05  level.  The  small  numbers 
involved  in  the  individual  correlations  do  not  provide  the  stability  desired 
for  a  hiqh  level  of  confidence  in  the  results.  The  one  clear  point  from  these 
data  is  the  tremendous  variability.  Only  one  shows  a  siqnificant.  positive 
correlation,  and  it  is  offset  by  a  siqnificant  nenat.ive  correlation.  These 
data  fail  to  support  the  PI  as  a  reliable  indicator  of  individual 
productivity. 

e.  While  the  branch  by  branch  rank  order  correlations  of  CM  and  PI 
reveal  that  there  is  no  stronq  positive  relationship,  the  fact  that  the  number 
of  pairs  in  each  correlation  is  small  may  mask  a  low  order  trend.  To  test  for 
this  possibility,  CM  ratinqs  (1  to  10)  were  used.  Within  each  division,  all 
CM  ratinqs  were  listed  from  hiqh  to  low  and  divided  into  three  catenories. 

The  rationale  was  that  these  confidential  supervisory  ratinns  could  be 
combined  within  a  division  and  accurately  identify  three  relative  levels  of 
performance  -  hiqh,  medium,  and  low.  For  each  division  the  ratinqs  were 
examined  for  cut  points  which  would  yield  three  approximately  equal  qroups. 
This  procedure  resulted  in  a  division  by  division  cateqori zation  that  varied 
somewhat  amonq  the  divisions  because  the  branch  chief  ratinqs  were  not  a 
forced  distribution. 

f.  After  the  three  .iob  performance  catenories  were  established  for 
each  division,  the  PI  rank  order  for  the  division  was  used  to  form  three 
catenories  with  the  same  number  of  individuals,  hut  usinq  PI  rank  order 
instead  of  CM  ratinqs.  For  example,  the  CM  ratinqs  for  one  division  resulted 
in  six  individuals  in  the  hiqh  performance  cateqory.  If)  in  the  medium,  and 
five  in  the  low  cateqorv.  These  ?1  persons  were  then  rank  ordered  from  hiqh 


TABLE  4-11 


BRANCH  CHIEF  CONFIDENTIAL  RATINGS  OF  TRASANA  ANALYSTS 


RATING 

N 

% 

10  (HIGHEST) 

21 

12.40 

9 

24 

14.20 

8 

22 

13.02 

7 

35 

20.71 

6 

30 

17.75 

5 

26 

15.38 

4 

4 

2.37 

3 

6 

3.55 

2 

1 

0.59 

1  (LOWEST) 

- 

- 

TOTAL 

169 

a 

99.97 

aTotal  does  not  equal  100%  due  to  rounding. 


to  low  using  their  individual  PI.  The  top  six  were  put  in  the  high  PI 
category,  the  next  10  in  the  medium,  and  the  last  five  in  the  low  category. 
The  two  classifications,  one  by  CM  and  one  by  PI,  were  then  compared  to  see 
how  different  the  two  groups  were.  Figure  4-6  shows  the  percentage  of 
individuals  by  division  whose  classification  by  PI  rank  order  was  different 
than  by  CM  rating.  The  percentage  mi sclassi fied  by  PI  varies  from  45  to  80 
percent.  The  misclassifications  did  not  exhibit  any  detectable  pattern. 
Several  individuals  moved  from  one  to  the  other  extreme  category. 1°  These 
data  show  that  the  individual  PI  does  not  accurately  categorize  individuals 
into  high,  medium,  and  low  performance  groups. 


l^The  data  for  this  analysis  are  not  provided  as  agreed  with  branch  chiefs. 
The  PI  categories  showed  marked  variance  from  the  CM  categories,  and  the 
movement  of  individuals  was  not  minor.  In  fact,  16.7%  of  the  individuals 
went  from  the  high  to  low  group  or  vice  versa,  and  a  total  of  58%  changed 
categories. 


i  CORRELATION  1  !  REOIJIREO  FOP  I  STATISTICAL 

|  COFFF IC IFNT  I  N  |  SIGNIFICANCE  AT  i  S I  Of!  I F I CANCE 
|  |  !  .OS  LEVEL  I 


4.2.3  Individual  PI  Difficulties 


Conceptually,  the  PI  for  an  individual  is  the  °S  modified  by  supervisory 
judgment  to  better  reflect  the  individual's  level  of  contribution  to  the  PS. 
This  adjustment  takes  the  form  of  increasing  or  decreasing  the  PS  by  up  to  30 
percent  for  the  individual. 

a.  One  real  difficulty  with  quantifying  the  individual  contributions 
to  the  PS  is  that  the  PS  actually  reflects  much  more  than  individual  effort 
and  therefore  so  does  the  PI.  The  PS  is  essentially  a  team  score.  Many  of 
the  tasks  in  a  project  are  actually  a  team  effort  which  are  virtually 
impossible  to  divide  into  individual  contributions.  Furthermore,  the  team 
performance  is  related  to  many  things  that  any  given  individual  on  the  team 
may  have  very  little  or  no  say  so  in,  such  as  who  is  on  the  team,  how  the 
project  is  planned  and  executed,  project  difficulty  (and  other  risk  factors), 
and  project  leader  decisions. 

b.  The  adjustment  factor  itself  is  part  of  the  problem.  Plus  or  minus 
30  percent  of  a  low  score  may  still  be  a  low  score,  and  the  opposite  in  the 
case  of  a  project  scoring  very  well.  A  high  performer  on  a  Doorly  planned  and 
executed  project,  or  a  poor  performer  on  a  very  well  managed  project  will  have 
Pi's  which  do  not  likely  reflect  the  true  difference  in  performances.  This 
situation  is  further  complicated  by  the  fact  that  the  supervisor  is  playing  a 
zero-sum  game.  To  give  someone  more,  someone  else  gets  less,  because  the 
total  will  always  be  the  same.  The  only  discretion  is  in  distribution  of  the 
available  points.  There  was  a  lot  of  variation  in  how  this  was  handled.  The 
inconsistency  with  which  supervisors  approached  this  situation  was  a  major 
part  of  the  PI  problem.  Some  supervisors  tried  to  make  the  Pi's  reflect  their 
judgments,  and  others  made  awards  almost  entirely  on  manpower  charges  without 
applying  the  individual  contribution  adjustment  factor. 

c.  The  same  inconsistency  applied  to  the  use  of  discretionary 
productivity  points.  The  lack  of  understanding  of  the  potency  of  DPP's  was 
evident  in  some  cases,  which  is  why  some  Pi's  were  unrealistically 
exaggerated.  It  has  already  been  pointed  out  that  the  top  nine  scorers  would 
not  be  top  scorers  without  bonus  awards.  There  is  no  problem  per  se  in  this 
fact.  The  problem  is  that  the  lack  of  consistency  makes  the  winners  and 
losers  too  dependent  on  factors  other  than  performance. 

d.  A  supervisor  has  at  his  disposal,  the  power  to  substantially  alter 
an  individual's  PI  by  the  application  of  the  +  30  percent  rule,  and  the  award 
of  DPP,  if  he  chooses  to  do  so.  The  fact  that  supervisors  were  so  inconsis¬ 
tent  in  this  regard  made  the  first  year  Pi's  meaningless.  Furthermore,  the  PI 
does  not  capture  all  of  the  relevant  aspects  of  job  performance  in  all  cases, 
and  reflects  factors  over  which  the  individual  has  no  control.  The  indivi¬ 
dual's  PI  is  therefore  subject  to  considerable  misinterpretation  and  abuse. 
Since  it  is  not  a  good  individual  performance  indicator,  and  there  is  a 
performance  appraisal  svstem  designed  to  document  job  performance,  the 
individual  PI  should  be  eliminated.  Its  retention  weakens  the  productivity 
measurement  system,  but  yet  does  not  provide  a  useful  performance  appraisal 
device. 
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4.2.4  Branch  Pi's 


a.  Branch  Pi's  ranged  from  1.17  to  1.81  with  a  mean  of  1.48  and  a 
standard,  ^deviation  of  0.15.  The  scoring  data  by  branch  are  in  table  C - 1 2  in 

-appendix  C.  The  branch  with  the  lowest  PI  had  less  than  one  TMM  charged  to 
scored  projects  during  the  scoring  period. 

b.  There  was  a  wide  range  of  charges  to  overhead  (OH)  among  the 
branches.  The  lowest  charge  was  0.5  TMM  and  the  highest  overhead  charge  was 
50.34  TMM.  The  effect  of  overhead  charges  on  branch  Pi’s  is  illustrated  in 
table  4-13.  The  effect  OH  has  on  any  given  PI  depends  on  several  factors  -- 
the  rules  used  in  charging  or  not  charging  overhead,  the  total  branch  TMM 
charged  to  scored  projects,  and  the  fact  that  some  branches  have  a  mission 
requiring  overhead  activities.  The  erratic  overhead  charging  pattern  and 
tendency  to  hide  actual  overhead  in  project  manpower  charges  dilutes  the 
usefulness  of  the  PI  for  branch  comparisons. 

c.  The  inclusion  of  overhead  in  the  PI  has  a  substantial  effect.  The 
value  of  what  a  branch  produces  is  not  lowered  by  overhead,  but  the  PI  is. 

The  reason  for  including  overhead  in  the  PI  is  to  keep  branches  from  charging 
time  to  overhead  instead  of  to  projects  to  affect  the  R  score.  The  result  is 
that  overhead  is  not  being  accurately  documented,  and  since  R  is  not  being 
controlled,  overhead  is  being  charged  to  projects.  Overhead  charging  has  to 
be  controlled,  but  including  it  as  a  PI  decrement  makes  the  organizational  PI 
less  accurate. 

4.2.5  Division  Pi's 

Division  Pi's  ranged  from  1.32  to  1.60.  Table  4-14  shows  the  division  Pi's 
with  and  without  OH,  and  reveals  the  same  difficulty  of  inconsistent  OH 
charging  as  the  branch  figures.  Division  OH  charges  ranged  from  11.10  to 
108.16  TMM.  More  complete  scoring  data  by  division  are  in  table  C-13  in 
appendix  C. 

4.3  TPMS  MANPOWER  COSTS 

There  is  no  way  to  determine  exactly  how  much  manpower  was  expended  on  TPMS 
during  the  first  scoring  year.  Some  of  the  things  required  by  TPMS  are  also 
required  for  good  management,  such  as  a  detailed  project  plan.  Such  activi¬ 
ties  cannot  legitimately  be  considered  TPMS  costs.  Activities  such  as  calcu¬ 
lating  Pi's  and  distributing  PP's  are  pure  TPMS  costs.  A  separate  manpower 
charging  code  was  established  for  documenting  time  spent  for  TPMS  activities. 
In  addition  to  the  above  mentioned  calculations,  time  spent  in  PMB  activities 
was  also  charged  to  TPMS.  Table  4-15  shows  the  TMM  charged  to  the  TPMS  over¬ 
head  account  during  the  individual  scoring  year  as  well  as  to  PRB  activities. 
As  mentioned  earlier,  the  PRB  existed  before  TPMS.  The  PRB  charges  are  shown 
separately  because  it  represents  a  substantial  investment  and  is  an  important 
part  of  TPMS,  even  though  it  is  not  a  pure  TPMS  cost.  It  should  be  noted  that 
most  of  the  first  year  PI  calculations  were  performed  after  the  end  of  June, 
and  therefore  are  not  reflected  in  the  figures  in  the  table. 


TABLE  4-13 


BRANCH  PI'S  CALCULATED  WITH  AND  WITHOUT  OVERHEAD  CHARGES 


REGULAR 

PI  WITH 

PI  w/o 

BRANCH 

OVERHEAD 

OVERHEAD 

OVERHEAD 

I  1 

4.06 

1.81 

i 

1.86  1 

2 

6.75 

1.64 

1.69 

i  3 

3.02 

1.63 

1.72 

4 

0.50 

1.63 

1.72 

5 

7.80 

1.62 

1.70 

6 

2.71 

1.59 

1.61 

7 

2.68 

1.58 

8 

7.35 

1.58 

1.68  ! 

9 

9.44 

1.57 

1.65  | 

10 

5.70 

1.55 

1.66  ! 

11 

16.63 

1.52 

1.71  | 

12 

16.72 

1.79  | 

13 

13.72 

KStfl 

1.75  ; 

14 

9.01 

1.44 

1.58 

15 

33.07 

1.42 

1.72 

16 

4.03 

1.42 

1.92 

17 

4.55 

1.41 

1.46 

18 

3.56 

1.38 

1.45 

19 

44.14 

1.36 

1.78 

20 

11.44 

1.33 

1.53 

21 

50.34 

2.38 

22 

12.70 

1.27 

1.39 

23 

13.12 

1.23 

1.40 

24 

4.96 

1.17 

1.83 

MEAN 

12.00 

1.48 

1.69 

SD 

12.86 

0.15 

0.20 

NOTE:  Branches  are  arranged  from  highest  to  lowest  PI 
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4.4  FIRST  YEAR  SCORING  PROBLEMS 

There  were  numerous  instances  of  computational  inaccuracies  and  questionable 
scoring  procedures  in  the  first  year  scoring  data.  There  were  many  reasons 
for  these  problems.  The  fact  that  it  was  a  new  system  was  certainly  a  large 
part  of  the  problem,  but  by  no  means  all  of  it.  The  record  keeping  and 
calculation  requirements  are  cumbersome  and  had  to  be  done  manually.  Such  a 
situation  invites  errors.  Furthermore,  the  diligent  monitoring  of  so  much 
data  kept  by  several  different  offices  would  approach  a  full-time  job.  While 
some  of  the  problems  were  due  to  system  start-up,  there  is  no  reason  to  assume 
the  second  year  data  will  be  much  improved  unless  considerable  simplification, 
automation,  and/or  monitoring  is  undertaken.  Selected  examples  illustrative 
of  the  kind  of  problems  that  were  not  uncommon  are  cited  below. 

4.4.1  Example  1  -  Situation  Not  Covered  by  TPMS  Guidance 

This  person  received  two  di scretionary  awards  consisting  of  1.57  and  2.38  PP. 
No  other  TMM  or  PP  were  recorded  for  this  individual.  Since  di scretionary 
awards  increase  an  existing  PI,  and  this  person  had  no  PI,  there  was  a 
situation  where  the  routine  application  of  the  formula  did  not  work.  In  this 
situation,  a  rather  innovative  solution  was  applied.  The  person's  PI  was 
computed  as  the  average  of  the  discretionary  awards  weighted  by  total  project 
TMM,  resulting  in  a  PI  of  1.78.^ 

4.4.2  Example  2  -  Inaccurate  Record  Keeping/Computation 

The  project  summarized  in  table  4-16  illustrates  record  keeping  and  computa¬ 
tion  difficulties.  Values  appearing  in  the  first  row  of  the  table  were  taken 
from  the  TPMS  project  log  (TMM  and  PP).  The  remaining  data  in  the  table  come 
from  the  TPMS  scoring  sheets.  While  the  two  sources  agree  that  the  expended 
TMM  was  23.07  by  the  responsible  division,  the  number  of  productivity  points 
earned  is  different  from  the  two  sources.  The  scoring  sheets  show  that  1.04 
more  PP  were  awarded  than  the  project  log.  Two  branches  charged  time  to  this 
project  which  summed  to  24.54  TMM.  This  is  more  than  1.0  TMM  over  the  value 
for  expended  TMM  in  the  project  log.  If  the  larger  expenditure  were  in  fact 
correct,  the  R  value  would  drop  from  0.99  to  0.96,  and  the  project  score  would 
change  from  1.37  to  1.32.  Also,  the  sum  of  PP  awarded  at  the  branch  level  was 
32.796,  which  is  inconsistent  with  both  the  division  score  sheet  and  the 
project  log. 

4.4.3  Example  3  -  System  Manipulation 

a.  This  example  illustrates  a  TPMS  design  difficulty.  The  scoring  of 
one  branch  with  regard  to  this  project  is  shown  in  table  4-17.  Again  the 
values  in  the  first  row  come  from  the  project  log  and  the  other  data  from  the 


H  If  the  PI  had  been  calculated  using  the  individual's  regular  OH  it  is  likely 
that  the  PI  would  have  been  much  higher.  Suppose  for  illustrative  purposes 
that  the  person  had  l  man-month  of  regular  OH.  The  PI  computed  with  this  OH 
is  4.95. 
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TABLE  4-16.  TPMS  SCORING  SHEET  SUMMARY  FOR  PROJECT  NUMBER  7 


TMM  EXPENDED 

PRODUCTIVITY  POINTS 

PI 

Project 

23.07 

31.60 

m 

Division 

23.07 

32.64 

1.41 

Branch 

24.54 

32.796 

X 

8.26 

10.98 

1.33 

Y 

16.28 

21.816 

1.34 

TABLE  4-17. 

TPMS  SCORING  SUMMARY  FOR  PROJECT  NUMBER  33 

TMM  EXPENDED 

PRODUCTIVITY  POINTS 

pi 

Project 

37.26 

61.86 

1.66 

Division 

37.17 

61.70 

1.66 

Branch 

24.63 

46.45 

1.89 

Person  A 

10.80 

24.24 

2.24 

Person  B 

4.21 

9.45 

2.24 

Person  C 

0.06 

0.14 

2.33 

Person  Total 

15.07 

33.83 

NA 

scoring  sheets.  The  branch  received  a  judgmental  award  of  5.57  PP  for  a  PI  of 
1.89  on  this  project.  Score  sheets  for  the  branch  show  only  three  individuals 
charging  to  this  project.  These  three  individuals  do  not  account  for  9.56  of 
the  man-months  charged  by  the  branch  nor  12.62  of  the  productivity  points 
awarded  to  the  branch.  The  assumption  is  that  these  9.56  man-months  were 
charged  by  the  branch  chief  who  then  took  a  minimum  negative  judgmental  award 
(judgmental  factor  of  0.70)  with  absolutely  no  loss  in  the  branch  PI  (which  of 
course  is  the  branch  chief's  PI).  This  negative  judgmental  award  to  the 
branch  chief  allowed  positive  judgmental  awards  to  the  three  individuals  in 
the  branch.  These  awards  put  two  of  the  three  individuals  into  the  top  nine 
scorers  in  the  activity  (PI  _>  2.00)  even  though  the  project  score  was  1.66, 
which  is  below  the  mean  activity  PS  of  1.68. 
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b.  The  problem  illustrated  in  this  example  is  that  when  a  branch  chief 
charges  to  a  project,  the  points  which  he  earns  may  be  used  as  bonus  points 
for  other  individuals  in  the  branch.  They  are  of  no  use  to  the  branch  chief, 
and  can  thus  be  distributed  to  other  team  members.  It  is  to  the  team’s 
advantage  then  to  have  the  branch  chief  cnarge  time  to  projects.  When  this 
situation  does  occur,  it  creates  ineqjity  for  individuals  on  projects  to  which 
the  branch  chief  does  not  charge  time,  or  on  projects  with  large  numbers  of 
participants  to  share  the  bonuses.  The  same  kind  of  manipulation  occurred  in 
instances  where  team  members  departed  the  organization  before  PP's  were 
di stri buted . 

4.5  PRODUCT  REVIEW  BOARD  ( PRB) 

Data  concerning  the  PRB  functioning  came  from  the  first  year  scoring  results 
and  interviews  with  division  chiefs,  branch  chiefs,  and  PRB  members. 

a.  The  results  of  the  Q  score  analysis  are  presented  in  4.1.4. 
Essentially,  the  Q  score  distribution  was  statistically  a  better  distribution 
than  those  for  the  other  factors.  What  is  not  revealed  by  the  distribution  is 
the  accuracy  of  scores,  or  how  well  the  Q  score  captures  actual  differences  in 
project  quality.  This  element  was  addressed  in  the  interviews. 

b.  The  result  of  the  interviews  wac  a  unaminous  lack  of  confidence  in 
the  Q  scores.  There  was  no  difference  in  responses  by  those  who  had  and  had 
not  fared  well  in  their  Q  scores.  It  is  generally  believed  in  the  organiza¬ 
tion  that  the  PRB  grades  projects  primarily  on  such  factors  as  report  format 
and  editorial  considerations.  Not  one  supervisor  was  found  who  felt  the  Q 
score  adequately  di scrimi nated  between  project  quality.  This  lack  of  confi¬ 
dence  in  the  score  given  by  the  PRB  is  a  substantial  detriment  to  morale. 
Several  branch  chiefs  expressed  concern  that  reports  were  being  tailored  more 
and  more  to  the  PRB  and  therefore  less  to  the  proponent.  Another  concern 
which  frequently  surfaced  was  that  larger  projects  are  penalized  because  it  is 
much  more  difficult  to  organize  and  write  a  readable  report  for  large 
complicated  projects. 

c.  As  was  pointed  out  in  4.1.4,  the  PRB  has  a  very  difficult,  if  not 
impossible,  task  in  trying  to  make  fine  discriminations  among  very  diverse 
projects  and  products.  The  major  problem  is  seen  as  the  grade  and  the  grading 
process,  not  the  PRB.  There  was  general  agreement  (incidently,  supported  by 
first  year  data)  that  the  PRB  is  necessary,  but  its  current  power  and 
influence  in  the  TPMS  scoring  process  is  not  well  received.  A  more  definitive 
balance  of  power  is  needed  between  the  responsible  division  (or  line 
management)  and  the  PRB  (staff  function). 

4.6  PRODUCTIVITY  MEASUREMENT  BOARD  (PMB) 

Data  for  the  analysis  of  PMB  functioning  came  from  the  first  year  scoring 
results,  interviews  with  PMB  members,  and  observing  one  PMB  meeting.  The  PMB 
is  responsible  for  reviewing  all  project  plans  for  accuracy  in  estimated 
resources  and  schedules.  The  primary  emphasis  in  this  analysis  was  how  the 
PMB  members  made  these  judgments,  and  how  effective  they  were  in  controlling 
the  input  estimates. 
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a.  First  year  scoring  results  reported  in  4.1.2  document  the  fact  that 
the  resource  estimates  were  not  adequately  controlled  the  first  year. 
Interviews  with  PMB  members  (division  chiefs)  revealed  that  the  general 
opinion  of  the  PMB  members  was  that  they  did  not  know  how  to  judge  the 
validity  of  estimates  from  divisions  other  than  their  own.  In  fact, 
considerable  frustration  was  expressed  in  regard  to  being  asked  to  make  such 
judgments.  As  a  result  of  this  uncertainty,  the  tendency  was  to  only  raise 
questions  in  cases  where  blatant  inaccuracies  were  suspected,  which  was  rather 
infrequent. 

b.  In  many  respects,  the  PMB  members  are  faced  with  the  same  dilemma  as 
the  PRB  members,  in  that  no  matter  how  hard  they  try,  it  is  not  clear  that 
their  task  is  reasonably  achievable.  Considering  that  over  100  projects  were 
entered  into  the  system  the  first  year,  and  that  60  to  90  reports  are 
generated  by  TRASANA  every  year,  the  magnitude  of  the  problem  comes  more 
clearly  into  focus.  In  addition  to  the  number  of  projects,  there  is 
considerable  variation  in  what  the  projects  involve.  There  is  just  not  an 
adequate  amount  of  time  for  division  chiefs  to  acquire  and  process  detailed 
information  about  each  and  every  project  and  keep  abreast  of  their  own 
division  responsibilities. 

4.7  TPMS  ALGORITHM  ANALYSIS  RESULTS 

a.  Numerical  error  analysis  revealed  that  the  numerical  error  associated 
with  both  the  project  score  and  the  PI  is  0.027.  The  contributing  factors  are 
errors  in  the  scoring  factors  and  the  computational  methods.  Depending  on  how 
the  PI  is  used,  this  may  be  an  unacceptably  high  numerical  error.  For 
example,  if  a  PI  is  computed  as  1.689  (first  year  mean)  the  true  value  could 
vary  from  as  little  as  1.662  to  as  much  as  1.716.  At  the  top  end  of  the 
scale,  a  PI  of  2.00  could  have  a  true  value  varying  from  1.973  to  2.027. 

b.  One  of  the  stated  reasons  for  using  a  multiplicative  rather  than  an 
additive  scheme  for  assembling  the  project  scoring  factors  is  that  an  additive 
scheme  is  compensatory  while  a  multiplicative  scheme  is  not.  In  fact,  a 
multiplicative  scheme  is  compensatory  to  a  certain  degree.  This  necessitates 
adopting  a  minimally  acceptable  value,  sometimes  called  a  critical  hurdle,  for 
Q  (currently  1.3).  The  resulting  scoring  function  is  a  combination  of  a 
compensatory  and  conjunctive  (critical  hurdle)  scheme. 

c.  The  use  of  distinct  ratio  efficiency  factors  (R  and  C)  which  are 
weighted  differently  is  intended  to  reflect  efficient  resource  use  and  also  to 
encourage  substituting  manpower  with  less  expensive  computer  usage.  However, 
the  different  weighting  distorts  actual  resource  costs  and  the  use  of  distinct 
factors  could  actually  encourage  replacing  some  computer  usage  by  wore 
expensive  manpower  to  maximize  the  project  score. 

d.  The  hierarchical  distribution  of  PP  (first  to  divisions,  then  to 
branches,  and  finally  to  individuals)  constrains  the  awarding  process  to  the 
extent  that  supervisors  may  not  be  able  to  award  PP  equally  for  comparable 
work  in  some  cases. 
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e.  Branch  and  division  Pi's  are  weighted  averages  of  PP  (including 
overhead)  over  a  scoring  period.  Scored  projects  can  carry-over  from  one 
period  to  the  next  while  overhead  would  not.  This  could  result  in 
organizations  having  equal  productivity  over  two  years,  but  their  Pi's  for 
each  year  would  be  different. 

4.8  RESPONSE  TO  SECOND  OBJECTIVE 

4.8.1  Objective  2:  To  determine  the  degree  to  which  TPMS  is  a  meaningful 

measure  of  productivity. 

a.  Study  Question  1:  To  what  extent  do  TP^s  ratings  agree  with 
confidential  supervisory  assessments  for  individuals  and  branches? 

(1)  Individuals.  There  was  not  agreement  between  individual  Pi's 
and  confidential  supervisory  assessments.  Furthermore,  the  individual  PI  is 
to  some  extent  subjective,  reflects  factors  beyond  the  individual's  control, 
and  is  subject  to  mi sinterpretation  and  abuse.  The  individual  PI  was  plagued 
with  so  many  difficulties  that  even  with  substantial  modifications,  its 
usefulness  will  be  questionable. 

(2)  Branch.  On  the  branch  level,  the  evaluation  was  not  based  on 
supervisory  evaluations.  Branch  Pi's  (as  well  as  division  Pi's)  were  much 
more  meaningful  than  individual  Pi's.  The  most  useful  aspect  of  the  PI  is  for 
organizational  units.  If  input  estimates  are  better  controlled  (see  4.1.2), 
and  the  problems  with  the  quality  score  (see  4.1.4)  and  overhead  (see  4.2.4) 
are  resolved,  the  organizational  PI  will  be  a  solid  productivity  indicator, 
especially  over  a  several  year  period. 

b.  Study  Question  2:  Does  TPMS  have  generally  the  same  impact  on  the 
different  TRASANA  organizational  divisions? 

No.  The  major  problem  here  was  the  treatment  of  overhead.  There  were 
considerable  inconsistencies  in  the  way  overhead  was  charged.  As  TPMS  is 
currently  structured,  organizational  units  are  punished  (PI  is  lowered)  for 
charging  time  to  overhead.  When  Pi's  are  calculated  without  overhead,  the 
resulting  branch  (and  division)  rank  order  is  virtually  uncorrelated  with  the 
complete  (with  overhead)  PI  rank  order.  The  inclusion  of  overhead,  which 
earns  productivity  points  at  a  lower  rate  than  the  worst  scoring  project, 
punishes  those  organizational  units  with  an  overhead  mission,  and  dilutes  the 
usefulness  of  the  organizational  PI. 

c.  Study  Question  3:  What  are  the  functional  relationships  among  the 
scoring  algorithm  variables? 

(1)  The  most  important  driving  factor  behind  first  year  scores  was 
the  manpower  utilization  factor  (R).  The  priority  (P)  and  computer 
utilization  (C)  factors,  except  for  a  very  few  C  scores,  were  constants  in  the 
scoring  equation.  The  timeliness  factor  (T)  equalled  1.0  for  over  half  of  the 
projects.  The  quality  (Q)  score  had  a  lesser  effect  on  the  project  score  than 
the  R,  C,  and  T  factors  because  of  its  range,  its  scoring  distribution,  and 
the  mul tipi icative  scoring  function. 


(2)  A  conservative  estimate  of  the  numerical  error  associated  with 
both  the  project  score  and  the  PI  is  0.027.  Depending  on  how  the  PI  is  used, 
this  may  be  an  unacceptably  high  numerical  error. 

(3)  The  use  of  distinct  ratio  efficiency  factors  (R  and  C)  which 
are  weighted  differently  could  actually  encourage  replacing  computer  usage  by 
more  expensive  manpower  to  maximize  the  project  score. 

(4)  The  hierarchical  distribution  of  PP  (first  to  divisions,  then 
to  branches,  and  finally  to  individuals)  constrains  the  awarding  process  to 
the  extent  that  supervisors  may  not  be  able  to  award  PP  equally  for  comparable 
work  in  some  cases. 

(5)  Branch  and  division  Pi's  are  weighted  averages  of  PP 
(including  overhead)  over  a  scoring  period.  Scored  projects  can  carry-over 
from  one  period  to  the  next  while  overhead  would  not.  This  could  result  in 
organizations  having  equal  productivity  over  two  years,  but  their  Pi's  for 
each  year  would  be  different. 

d.  Study  Question  4:  What  is  the  operational  impact  of  the 
Productivity  Measurement  Board  (PMB)  and  the  Product  Review  Board  (PRB)? 

(1)  PMB.  The  most  important  function  of  the  PMB  was  controlling 
input  estimates  for  R,  C,  and  T.  It  was  not  effective  at  this,  especially  in 
the  case  of  R.  This  does  not  reflect  poorly  on  the  PMB.  This  way  of 
controlling  estimates  was  a  good  idea  that  did  not  work. 

(2)  PRB.  The  PRB  appears  to  have  done  a  good  job  in  the  area  of 
quality  control  (28  percent  of  the  reports  were  rejected  the  first  time 
around),  but  the  actual  assigning  of  "grades"  was  less  successful.  The 
problem  was  that  the  PRB  tried  to  make  very  fine  distinctions  between  projects 
based  on  reviewing  reports  in  a  relatively  limited  amount  of  time. 
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CHAPTER  5 


SUMMARY  AND  CONCLUSIONS 

This  chapter  sumnarizes  the  findings  of  the  Phase  I  evaluation  of  TPMS 
relative  to  the  original  design  attributes  of  the  system,  and  presents  the 
conclusions  supported  by  the  analysis.  Summary  responses  to  the  objectives 
and  study  questions  are  at  the  end  of  chapters  three  and  four. 

5.1  DESIRED  ATTRIBUTES  OF  TPMS 

a.  Table  5-1  shows  the  results  of  the  first  year  evaluation  relative  to 
the  original  design  attributes  of  TPMS  (see  chapter  1). 


TABLE  5-1.  FINDINGS  RELATED  TO  TPMS  DESIGN  ATTRIBUTES 


ATTRIBUTE 

FINDINGS 

Relevant 

Yes;  Measures  productivity 

Rational 

Yes;  Except  individual  PI  and  DPP 

Fair 

Individual:  Input/output  mi smatch 
Organization:  Overhead  mission  di sadvantaged 

Affordable 

? 

Flexi ble 

Undetermi ned 

Simpl  e 

No 

Transparent 

No 

Durable 

TBD 

(1)  TPMS  was  judged  to  be  relevant  to  the  organization,  its 
products,  and  to  the  jobs  that  produce  the  products.  The  classical  definition 
of  productivity  is  value  of  output  over  value  of  input,  and  captures  both 
efficiency  and  effectiveness.  The  R  and  C  factors  of  TPMS  capture  efficiency, 
and  the  T  and  Q  factors  capture  effectiveness. 1  The  weakness  in  terms  of 
meeting  the  classical  definition  is  in  controlling  estimated  manpower,  which 
is  the  single  most  important  factor  in  establishing  project  worth  under  the 
cost-plus-incentive  concept. 


1 The  P  factor  captures  neither  efficiency  nor  effectiveness.  In  terms  of  the 
classical  definition  of  productivity,  the  P  factor  is  questionable .  It  was 
not  an  issue  in  this  analysis  because  it  reverted  to  a  constant  value  for  all 
projects. 
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(2)  The  system  was  judged  to  be  conceptually  sound,  and  therefore 
rational  ,  but  some  of  the  component  elements  did  not  work  as  well  as  others. 
Basically,  the  inconsistent  application  of  judgmental  and  di scretionary  pro¬ 
ductivity  points  had  an  undesirable  impact  on  individual  Pi's.  Furthermore, 
so  many  of  the  tasks  in  projects  are  actually  group  tasks,  or  team  tasks,  that 
there  is  no  objective  way  to  measure  the  relative  contributions  of 

i ndi vi dual s . 

(3)  The  most  frequently  stated  complaint  about  TPMS  was  that  it  is 
unfair.  There  is  some  basis  in  fact  for  this  complaint.  On  an  individual 
level,  there  is  considerable  potential  for  an  input  effort/output  reward  mis¬ 
match.  For  example,  two  persons  with  equal  capabilities  and  putting  forth 
equal  time  and  effort  on  two  equally  scoring  projects  (i.e.  equal  project 
scores)  may  not  earn  productivity  points  at  the  same  rate.  This  could  occur 
for  reasons  totally  beyond  the  individual's  control,  such  as  one  study  manager 
using  judgmental  awar's  to  reward  "super"  performers,  and  the  other  study 
manager  distributing  all  productivity  points  based  solely  on  manpower  charged 
(i.e.,  everybody  on  the  team  earns  at  the  same  rate).  On  an  organizational 
level,  those  branches  ana  divisions  with  an  overhead  mission  are  disadvan¬ 
taged  . 


(4)  No  judgment  is  offered  on  the  affordability  of  the  system 
because  that  is  a  decision  for  management  to  make  after  all  the  facts  are 
available. 

(5)  The  flexibility  of  the  system  was  not  determined.  One  problem 
related  to  flexibility  is  the  disadvantage  to  units  with  an  overhead  mission. 

(6)  The  system  is  not  simple.  The  PI  calculations  are  cumbersome 
and  complex,  and  the  staff  generally  does  not  understand  how  the  various 
factors  and  components  interact  to  affect  scores.  The  biggest  problems  here 
are  the  use  of  judgmental  and  discretionary  productivity  points  (and  their 
impact),  and  dividing  productivity  points  between  organizational  units. 

(7)  The  system  is  not  transparent.  Most  of  the  staff  do  not  see  a 
direct  relationship  between  input  effort  and  output  score  -  which  is  one 
reason  so  many  see  the  system  as  unfair. 

(8)  More  time  is  required  to  determine  the  durability  of  the  system. 
Part  of  the  durability  depends  on  lowering  resistance  to  certain  elements  of 
the  system  (addressed  in  chapter  6). 

b.  In  summary,  several  problems  were  found  by  the  evaluation,  but  the 
system  seems  to  be  conceptually  sound.  It  has  more  validity  on  the  organiza¬ 
tional  level  than  for  individuals,  and  probably  has  its  greatest  use  over  a 
period  of  several  years  so  that  short  term  fluctuations  are  averaged  out. 

5.2  CONCLUSIONS 


5.2.1  General 


a.  TRASANA  is  a  healthy  organization  with  high  morale,  job  satisfac¬ 
tion,  and  a  motivated  work  force  that  is  challenged  and  proud  of  the  organi¬ 
zation. 


b.  There  is  mixed  reaction  to  TPMS.  Supervisors  have  a  more  positive 
view  than  nonsupervisors,  but  the  majority  of  the  staff  have  legitimate 
concerns  about  TPMS  fairness,  and  how  TPMS  will  affect  the  creative, 
cooperative  work  environment. 

c.  TPMS  is  a  conceptually  sound  productivity  measurement  system,  but 
many  of  its  functional  elements  need  revision. 

d.  The  project  plan  is  a  very  positive  contribution  of  TPMS. 

e.  Firm  guidelines  are  needed  on  overhead  so  the  system  will 
accurately  document  the  level  of  overhead  use.  Different  categories  of 
overhead  are  needed,  especially  in  those  situations  where  there  is  an 
overhead  mission. 

5.2.2  Productivity  Index 


a.  The  individual  PI  did  not  perform  well  and  is  the  source  of 
considerable  staff  uneasiness.  Since  accurate  reflection  of  individual 
contributions  in  group  analysis  efforts  cannot  be  captured  with  an  objective 
measure,  the  system  would  be  more  accurate  and  less  objectionable  to  the 
staff  if  the  individual  PI  were  eliminated. 

b.  The  organizational  PI  would  be  greatly  strengthened  by  eliminating 
overhead  from  the  PI  calculation. 

c.  The  organizational  PI  will  be  a  solid  productivity  indicator  if  the 
input  estimates  are  more  accurate.  The  PMB  was  not  effective  in  controlling 
input  estimates. 

5.2.3  Product  Quality  Score 


a.  The  PRB  serves  a  necessary  quality  control  function. 

b.  The  quality  scoring  by  the  PRB  is  such  a  strong  morale  detriment 
that  it  should  be  changed. 

c.  PRB  scoring  procedures  and  authority  should  be  more  clearly  defined 
and  specified. 

5.2.4  Scoring  Algorithm 


a.  The  scoring  algorithm  is  unnecessarily  complicated. 

b.  Judgmental  and  discretionary  awards  caused  many  problems  and  should 
be  eliminated.  They  add  a  subjective  element  that  is  performance  appraisal 
oriented,  and  did  not  work  well.  Individual  performance  appraisal  and  team 
productivi ty  measurement  should  be  separate. 
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c.  Management  flexibility  is  required  to  intervene  in  situations  where 
c i rcumstances  beyond  the  team's  control  result  in  a  poor  score  -  such  as 
undertaking  a  very  risky  project. 

5 . 2 .  S  Follow-Up  Evaluation 

This  study  team  has  formed  definite  opinions  about  TPMS  and  has  made  substan¬ 
tial  recommendations  for  changes  in  chapter  6.  To  preserve  objectivity,  any 
further  TPMS  evaluation  should  be  performed  by  another  study  team.  Since  TPMS 
is  a  complex  measurement  system,  a  strong  background  in  human  measurement  and 
management  is  needed. 
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CHAPTER  6 


RECOMMENDATIONS 


6.1  PURPOSE 

This  chapter  contains  recommendations  for  improving  those  aspects  of  TPMS 
which  did  not  work  well  during  the  first  scoring  year. 

6.2  APPROACH 

The  major  problem  areas  are  listed  in  table  6-1  along  with  the  recommendations 
for  correcting  the  problems.  Specific  strategies  and  procedures  for  imple¬ 
menting  the  recommendations  are  presented  in  appendix  F  in  the  form  of  an 
alternative  TPMS  scoring  algorithm.  The  rationale  behind  each  of  the 
recommendati ons  is  discussed  below. 

6.3  RECOMMENDATIONS 

6.3.1  Estimated  Manpower  (Rp) 

a.  The  overestimation  of  manpower  was  the  largest  problem 

from  the  perspective  of  actually  measuring  productivity  in  the  classical 
sense.  Currently  a  division  chief  can  enter  a  project  into  TPMS  if  his 
projected  manpower  requirement  is  not  more  than  125  percent  of  available 
manpower  (125%  rule).  This  125  percent  should  be  reduced  to  110  percent  to 
encourage  more  accurate  estimating.  Furthermore,  it  may  need  to  be  reduced 
even  more  in  the  future.  The  extra  10  percent  is  recommended  for  now  because 
perfectly  accurate  estimating  is  not  realistic. 

b.  The  categorical  scoring  recommenaed  for  R,  T,  and  Q  wi 1 1  encourage 
more  accurate  manpower  estimating  by  not  rewarding  overestimation.  Further¬ 
more,  at  the  end  of  each  scoring  period  each  division's  accuracy  over  the 
period  can  be  determined,  and  a  bonus  given  to  the  most  accurate  estimator. 

The  bonus  could  take  the  form  of  increasing  the  division  PI  somewhat  (the 
closer  the  factor  score  to  1.0,  the  higher  the  bonus),  or  some  other  reward 
could  be  devised.  Since  punishment  is  generally  less  effective  than  reward 
for  effecting  long  term  behavior  change,  and  also  has  undesirable  side- 
effects,  rewarding  the  desired  results  is  the  favored  approach  over  punishing 
i naccuracy . 

6.3.2  Productivity  Index  (PI) 


a.  Individual.  In  its  current  form,  the  PI  for  individuals  does  not 
accurately  reflect  individual  contributions  and  has  considerable  potential  for 
abuse.  An  example  would  be  a  supervisor  who  accepts  the  PI  as  accurate  with¬ 
out  considering  its  limitations,  and  subsequently  writes  performance  apprais¬ 
als  based  on  an  "individual's  productivity"  using  the  PI.  For  this  reason, 
retention  of  the  individual  PI  is  not  recommended.  A  less  desirable  alter¬ 
native  is  to  eliminate  judgmental  and  di scretionary  awards  and  assign  every 
team  member  the  team  (project)  score.  The  judgmental  and  discretionary  awards 
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are  an  attempt  to  make  an  organizational  productivity  measurement  system  serve 
as  an  individual  performance  appraisal  device.  As  shown  in  the  first  year 
results,  this  just  did  not  work.  The  system  will  work  better  without  the 
performance  appraisal  aspect  (be  a  stronger  organizational  index),  and  there 
is  already  a  separate  personnel  appraisal  system  where  supervisors  can  reflect 
individual  job  performance. 

b.  Organizational.  On  the  organizational  level,  the  problem  is 
overhead.  One  approach  is  to  eliminate  overhead  from  the  PI  computation,  and 
to  have  a  penalty  for  abuse  of  overhead.  Another  approach  is  to  have 
different  earning  rates  for  overhead  (see  6.3.4). 

6.3.3  Quality  Scoring  (Q) 

The  PRB  serves  a  very  valuable  quality  control  function  which  should  be 
retained;  however,  there  are  several  difficulties  with  the  current  Q  scoring. 
Categorical  scoring  (excellent,  okay,  revision  required)  will  help  consider¬ 
ably,  but  that  is  not  enough.  A  checks-and-bal ance  procedure  is  needed  to 
avoid  the  possibility  of  the  PRB  exerting  too  much  influence  over  reports. 

The  primary  responsibility  for  reports  should  rest  with  line  management.  The 
PRB  should  be  required  to  complete  a  checklist  and  rating  scale  for  each 
report  which  is  promptly  furnished  to  the  responsible  division  chief.  This 
checklist  should  contain  those  factors  the  board  uses  in  determining  the 
quality  score.  It  is  likely  that  these  factors  are  to  a  large  extent  report 
organization  and  readability.-^  Additionally,  accountability  is  a  two  way 
street.  Permanent  PRB  members  (voting  members)  should  be  held  accountable  for 
their  productivity  by  being  scored  on  their  thoroughness,  helpfulness,  and 
fairness  by  project  study  managers,  branch  chiefs,  and  division  chiefs 
(instances  were  reported  of  PRB  members  not  having  read  the  report  prior  to 
the  board  meeting).  A  simple  one-page  rating  form  for  this  purpose  could  be 
developed  rather  easily. 

6.3.4  Overhead 


Guidelines  for  charging  overhead  are  essential  if  this  problem  is  to  be 
overcome.  The  guidelines  should  establish  "acceptable"  overhead  limits.  For 
example,  supervisors  are  generally  overhead,  and  there  are  legitimate  overhead 
activities  that  take  up  a  small  amount  of  everyone's  time.  This  is  widely 
recognized  in  such  activities  as  applying  for  research  grants,  where  a  figure 
around  30  percent  is  fairly  typical  of  expected  research  overhead  expense. 

The  first  year  treatment  of  overhead  diluted  PI  usefulness. 


1  The  Colonel  who  sewed  as  PRB  board  president  during  the  first  scoring  period 
was  interviewed  before  departing  TRASANA .  He  reluctantly  admitted  that  this 
was  the  case  even  though  he  "wished"  it  could  different.  He  had  no 
recommendations  for  how  to  change  the  situation .  The  problem  is  that  PRB 
members  do  not  have  enough  time  to  gather  and  process  detailed  information 
on  each  and  every  project.  The  possible  return  on  the  investment  of  having 
them  take  such  time  would  not  likely  be  worth  it. 


6.3.5  Scoring  Algorithm 


"1 

If  three  categories  are  used  for  the  R,  T,  and  0  factors,  the  current 
algorithm  can  be  replaced  by  27  possible  outcomes  without  loss  of  information, 
and  with  reduced  record  keeping  requirements  (especially  if  individual  PI  is 
discontinued).  These  27  possible  outcomes  should  be  evaluated  and  weighted  in 
advance,  so  a  project  score  is  automatical ly  assigned  based  on  the  outcome 
category  (see  appendix  F  for  details). 

6.3.6  Risk  Taking 

There  is  considerable  concern  that  TPMS  will  reduce  staff  willingness  to  take 
on  unknown  and  risky  projects.  Since  no  workable  risk  index  is  available, 
this  problem  can  be  handled  by  not  applying  TPMS  scoring  dogmatical ly. 

Management  should  reserve  the  perogative  of  adjusting  project  scores  where 
risk  resulted  in  inequitable  results. 


APPENDIX  A 


TPMS-TE  STUDY  PLAN 
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1.  PURPOSE 

To  evaluate  Phase  I  f  the  first  year)  of  the  three-year  :-st  i.npl  eT*>ntat  i  o-;  .•<’ 
the  TRASANA  Technical  Staff  Product  i  v  i  t  /  Measurement  S.y  -.:»*•"  (  rEM  -•  . 

?.  BACKGRiKJNi. 

a.  T°MS  ’  s  a  methodology  designed  to  measure  t‘-e  pr  ■  v  1  i  w  • t  he 
TRASANA  technical  staff,  i  is  organizational  elements,  a"c  <  r  <  ;  rr.ji  v  i  gu  i 
employees.  It  yields  a  single  dimensionless  index  as  the  mess, re 
productivity.  The  formulation  of  this  index  is  intended  to  he  consistent  wi¬ 
the  traditional  definition  of  productivity  as  value  of  output  divided  oy  vaiv 
of  input.  !n  principle,  TPMS  places  TRASANA  on  a  "fixed  price  plus  incentive 
basis  for  producing  its  various  products.  TPMS  was  designed  with  ‘be 
following  attributes  in  mind: 

(1)  Relevant  -  to  the  organization,  its  products,  and  t.o  the  jobs 
that  produce  the  products. 

(2)  Rational  and  fair  -  so  that  it  can  be  accepted  and  used  by  a 
sophisticated  workforce. 

(3)  Affordable  -  in  terms  of  direct  and  indirect/overhead  costs  for 
operation  and  maintenance. 

(4)  Flexible  -  to  couple  with  a  dynamic  workload,  organi zat ion ,  mrl 
operations. 

(5)  Simple  -  both  in  principle  and  practice. 

(6)  Transparent  -  so  as  to  facilitate  understandi ng  of  cause  and 
effect  relationships. 

(7)  Durable  -  to  support  multi-year  use. 

b.  Given  that  TPMS  enables  reasonably  equal  opportunity  for  productivity 
and  scoring  throughout  the  organization,  then  the  resulting  productivity  index 
(PI)  can  be  used  to  compare  the  relative  productivity  of  TRASANA  divisions, 
branches,  and  individuals.  Management  and  individual  goals  can  be  structured 
in  terms  of  changes  in  the  PI.  The  degree  to  which  the  goals  are  met  could  be 
a  factor  in  performance  appraisal.  The  TPMS  has  been  adopted  at  TRASANA  for  a 
three  year  trial  beginning  1  Oct  80.  This  study  is  the  first  phase  of  a  3 
year,  3  phase,  evaluation  of  the  TPMS  test  (TPMS-T). 

3.  PROBLEM 

To  determine  the  degree  to  which  TPMS  provides  a  meaningful  measure  of  the 
productivity  of  the  TRASANA  technical  staff  and  its  organizational  elements. 


4.  IMPACT  OF  PROBLEM 


The  purpose  of  the  TPMS  test  period  is  to  develop  experience  working  with  the 
system  so  its  effects  on  management  practices,  employee  motivation,  and 
productivity  can  be  judged.  A  systematic  collection  and  assimilation  of 
information  about  the  system  test  is  required  to  assess  these  factors. 
Implementation  of  an  untested  system  could  result  in  having  inaccurate 
productivity  results,  or  even  hamper  productivity  by  adversely  affecting 
management  practices  and  employee  motivation. 

5.  SCOPE 

a.  The  study  will  be  limited  to  evaluating  the  results  of  the  first  trial 
implementation  year  of  TPMS. 

b.  TPMS  was  developed  in  part  as  a  local  response  to  requirements  of  the 
new  Performance  Appraisal  (PA)  and  Merit  Pay  (MP)  systems.  While  informal 
integration  of  TPMS  data  and  PA  and  MP  systems  may  occur  during  the  test 
period,  this  evaluation  is  limited  exclusively  to  TPMS.  No  assessment  of  the 
PA  or  MP  systems  is  intended. 

c.  TPMS-TE  is  an  assessment  of  a  specific  productivity  measurement 
system.  It  is  not  intended  to  be  a  comprehensive  study  of  methods  to  measure 
TRASANA  or  research  and  development  type  productivity. 

d.  One  of  the  desired  characteristics  of  TPMS  is  that  it  be  affordable. 
Available  data  will  be  collected  on  resources  expended  on  the  first  year  test 
of  TPMS;  however,  judgments  on  whether  the  costs  are  a  worthwhile  investment 
are  beyond  the  scope  of  this  study. 

6.  OBJECTIVES  FOR  PHASE  I  EVALUATION 

a.  To  determine  how  the  TRASANA  technical  staff  (TTS)  perceives  TPMS. 

The  study  questions  for  this  objective  are: 

(1)  What  is  the  TTS  general  attitude  toward  TPMS? 

(2)  What  is  the  perceived  motivational  impact  of  TPMS? 

(3)  Is  TPMS  seen  as  a  viable  managerial  tool? 

b.  To  determine  the  degree  to  which  TPMS  is  a  meaningful  measure  of 
productivity.  The  study  questions  for  this  objective  are: 

(1)  To  what  extent  do  TPMS  ratings  agree  with  confidential 
supervisory  productivity  assessments  for  individuals  and  branches. 

(2)  Does  TPMS  have  generally  the  same  impact  on  the  different  TRASANA 
organization  divisions? 

(3)  What  are  the  functional  relationships  among  the  scoring  algorithm 
variables? 
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(4)  What  is  the  operational  impact  of  the  Productivity  Measurement 
8oard  (PMB)  and  Product  P.e.iew  Board  ( PRB ) ? 

c.  To  develop  a  TPMS  data  base  consisting  of  Phase  I  TPMS-T  results. 
Available  data  on  resource  imp1 ications  will  be  included.  This  data  base  will 
form  the  benchmark  for  comparison  of  future  results. 

7.  ASSUMPTIONS 

a.  Phase  I  of  the  TPMS-T  implementation  will  end  30  June  rail. 

b.  TPMS  will  regain  essentially  unchanged  during  Phase  II  of  t  e  test 
implementation  (test  integrity). 

c.  The  test  will  apply  equally  to  all  TRASANA  Technical  Staff  (e.g. 
civilian  and  military). 

d.  Key  organizational  supervisors  know  the  relative  long  term 
productivity  of  employees  they  supervise. 

e.  Division  chiefs,  branch  chiefs,  and  the  TTS  will  provide  complete, 
accurate  and  timely  information  for  the  study. 

f.  First  year  test  results  and  other  required  inputs  will  be  provided  on 
time,  including  manpower  and  computer  time  charges. 

g.  Management  will  respect  the  confidential  data  collected  as  a  part  of 
the  study. 

8.  CONSTRAINTS 

a.  Not  all  of  the  TTS  will  have  completed  TPMS  scored  projects  during  the 
first  test  implementation  year. 

b.  There  is  no  basis  for  comparing  test  period  productivity  with  past 
productiv i ty. 

9.  DATA  COLLECTION  METHODOLOGY 

The  study  methodology  consists  of  several  independent  data  collection 
steps  and  the  development  of  accompanying  data  collection  instruments. 

a.  TRASANA  Technical  Staff  Perception  Questionnaire. 

(1)  This  questionnaire  will  be  designed  to  gather  the  data  required 
to  answer  the  three  study  questions  supporting  the  first  objective:  general 
attitude  toward  TPMS,  perceived  motivational  aspects  of  TPMS,  and  the 
perceived  impact  of  using  TPMS  as  a  managerial  tool.  After  the  questionnaire 
is  constructed  it  will  be  tested,  evaluated  for  reliability,  and  revised  as 
appropriate  before  administering  to  the  TRASANA  staff. 


(2)  The  final  version  questionnaire  will  be  admi ni stered  to  the 
available  TTS  at  the  end  of  the  first  test  year,  but  before  final  first  year 
Pis  are  compiled  (July  1981).  This  timing  is  to  allow  maximum  exposure  to  the 
system,  but  to  gather  perceptions  before  the  first  year  results  have  had  a 
chance  to  impact  impressions. T  The  surveys  will  be  administered  on  a  group 
basis  to  minimize  interaction  of  staff  members  while  responding. 

b.  Productivity  Criterion  Measure.  Based  on  a  literature  survey  of 
research  and  development  productivity  factors,  a  short  scale  or  index  will  be 
devised  for  supervisory  personnel  (division  and  branch  chiefs)  to  use  in 
evaluating  technical  employee  productivity.  This  scale  or  index  will  be  used 
in  interviews  to  obtain  a  supervisory  assessment  of  productivity.  This 
assessment  will  be  conducted  privately,  and  without  the  individual  employee  or 
any  other  person  being  given  access  to  the  ratings  except  the  analysts 
assigned  to  this  project. 2  Division  chiefs  will  rate  branch  chiefs,  and 
branch  chiefs  will  rate  persons  in  their  branches.  Because  of  the  sensitivity 
of  this  information,  special  precautions  will  be  employed.  Ratings  obtained 
for  this  study  will  not  be  available  to  supervisory  staff,  will  not  be  placed 
in  personnel  files,  and  will  not  be  used  in  any  way  that  would  allow 
individual  identification. 

c.  Survey  of  Divisional  Impact.  Survey  questions  on  relevant  issues, 
such  as  changes  in  managerial  practices,  will  be  developed  and  distributed  in 
advance  to  division  chiefs.  Available  division  chiefs  will  be  interviewed  to 
obtain  answers  to  the  impact  questions  and  any  other  information  relevant  to 
the  TPMS  test  evaluation. 

d.  TPMS-T  first  year  results.  Project  scoring  sheets,  productivity 
points  (PP),  and  productivity  indices  will  be  collected  for  each  individual, 
branch,  and  division. 

e.  PMB  and  PRB  functioning.  Rating  forms  will  be  developed  for 
surveying  PMB  members  on  its  functioning,  and  to  survey  study  managers  on  the 
PRB  functioning.  All  available  PMB  members  will  be  surveyed,  and  study 
managers  who  have  completed  scored  projects  under  TPMS-T  will  be  sampled. 

10.  DATA  ANALYSIS  METHODOLOGY 

The  data  analysis  methodology  employed  to  answer  the  study  questions  and 
accomplish  the  objectives  are  now  described.  Table  1  presents  the  objectives 
a"d  study  questions  by  data  sources  and  principal  analytic  techniques. 

At  this  time  it  is  unknown  how  the  first  year  results  will  be  distributed  and 
what  emphasis  will  be  put  on  them  in  the  different  divisions  and  branches. 

To  avoid  the  possible  contaminating  effects  of  first  year  scores,  the 
perception  survey  will  be  administered  before  the  final  scores  are  available. 

* Currently  only  two  analysts  ore  involved  in  this  project.  A  third  analyst 
will  be  assigned  eventually  to  work  on  data  analysis.  It  is  not  anticipated 
that  the  third  analyst  will  have  access  to  this  information  until  it  is  coded 
in  such  a  manner  as  to  preclude  individual  identification. 


TABLE  1 


OATA  SOURCES  AND  ANALYTIC  TECHNIQUES  FOR 
ANSWERING  STUDY  QUESTIONS  AND  OBJECTIVES 


08JECTIVE/STUDY  QUESTION 

DATA  SOURCES 

ANALYTIC  TECHNIQUES 

I,  TTS  perceptions  of  TPMS 

(i) 

General  Attitude 

Q.  I 

Descriptive  statistics; 
inferential  statistics 

(2) 

Perceived  Motivational 

Q,  I 

to  determine  differences 

Aspects 

among  organizational 
units. 

(3) 

Perception  as  Managemert 
Tool 

Q,  I 

II.  Meaningful  measure  of 

productivity 

0) 

Relationship  between  PI 

C.  T 

Inferential  statistics 

and  supervisory  ratings 

including  correlation 
analysis. 

(2) 

Equivalent  divisional 

Q.  I. 

T 

Descriptive  statistics; 

Impact 

Inferential  statistics  to 
determine  differenres 
among  divisions. 

(3) 

Relationships  among  PI 

T 

Mathematical  description 

Variables 

of  the  scoring  algorithm. 

(4) 

Impact  of  PMB  and  PRB 

0,  I,  T, 

S.  B 

Descriptive  statistics. 

Q  -  Questionnaire 
I  -  Division  ohief  interviews 

T  -  TPMS  first  implementation  year  data  (PP,  PI,  project  eoore  sheets, 
project  plans ) 

C  -  Criterion  measure  (division  and  branch  chiefs) 

S  -  Study  manager  survey 
B  -  PMB  member  survey 
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a.  TTS  perceptions  of  TPMS.  This  includes  study  questions  on  general 
attitude,  motivational  aspects,  and  perceptions  as  a  management  tool.  Data 
from  the  TTS  questionnaire  and  division  chief  interviews  will  be  analyzed 
using  descriptive  statistics.  Inferential  statistics  will  be  used  to  test 
comparisons  by  organizational  units  and  demographic  character!' sties. 

b.  Relations  between  rankings  by  PI  and  rankings  by  supervisors. 
Inferential  statistics  including  correlation  analysis  will  be  used  to 
determine  the  relationship  between  ranks  of  individuals  by  the  PI  and 
supervi sors. 

c.  Impact  on  divisions.  Data  from  the  TTS  questionnaire,  division  chief 
interviews,  and  first  year  scoring  data  will  be  analyzed  using  descriptive 
statistics.  Comparison  of  TPMS  scores  by  divisions  will  be  made.  Inferential 
statistics  will  be  employed  to  test  the  significance  of  any  differences  among 
divi sions. 

d.  Relationships  among  the  scoring  algorithm  variables.  The  scoring 
algorithm  will  be  analyzed  mathematically.  The  relationships  among  the 
variables  will  be  described  and  their  relative  effect  on  the  PI  assessed. 

e.  Impact  of  PMB  and  PRB.  Relevant  data  from  the  TTS  questionnaire, 
division  chief  interviews,  first  year  scoring  results,  study  manager  survey, 
and  PMB  survey  will  be  analyzed  descriptively.  PMB  effectiveness  in  reviewing 
project  plans  and  critiquing  resource  estimates  will  be  assessed.  Likewise 
the  operational  impact  of  the  PRB  will  be  assessed. 

f.  Table  2  summarizes  the  project  tasks  including  data  collection  and 
data  analysis  procedures.  For  a  more  detailed  description  of  individual 
tasks,  see  the  Phase  I  Evaluation  of  TRASANA  Technical  Staff  Productivity 
Measurement  System  (TPMS)-Test  Project  Plan,  29  April  1981. 

11.  OPERATIONAL  SUPPORT  REQUIREMENTS 

a.  Deputy  Director  for  Technical  Operations 

(1)  Approve  project  plan  and  study  plan. 

(2)  Guarantee  confidential ity  of  sensitive  data. 

(3)  Maintain  TPMS-T  test  integrity. 

(4)  Provide  accessibility  of  study  team  to  division  chiefs,  branch 
chiefs,  TRASANA  technical  staff,  PMB  and  PRB  participants  and  procedures. 

(5)  Provide  support  in  the  coordination  and  execution  of  data 
collection  activities  (including  assembly  of  necessary  staff  for  surveys). 

(6)  Encourage  full  cooperation  of  all  TRASANA  supervisors  and 
technical  staff  with  the  TPMS-T  evaluation  effort. 
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I 


DEVELOP 

TPMS 

PERCEPTION 

QUESTIONNAIRE 


COLLECT  TPMS 
PHASE  I  SCORING 
DATA 


ADMINISTER 
QUESTIONNAIRE 
TO  TRASANA 
TECHNICAL  STAFF 


CODE  S 
ANALYZE 
DATA 


CODE  RESULTS;  COMPARE 

PREPARE  RESULTS  ORGANIZATIONAL 
SUMMARY  RESULTS 


TABLE  2 
TPMS-TE  TASKS 


PREPARE 

PROJECT 

PLAN 


PREPARE 

STUDY 

PLAN 


INTERVIEW  DIVISION 
CHIEFS  FOR  EVALUA¬ 
TION  OF  TPMS  DIVI¬ 
SIONAL  IMPACT 


ORGANIZE  AND 
ASSIMILATE  DIVI¬ 
SION  CHIEF  INTER¬ 
VIEW  DATA 


- j - 

DEVELOP 

TPMS  CRITERION 
MEASURE  (CM) 


ADMINISTER  CM 
TO  DIVISION 
AND  BRANCH 
CHIEFS 


CODE  RESULTS 
PREPARE  DESCRIPTIVE 
SUMMARY 


INTEGRATED  DATA 
ANALYSIS 


BRIEFING 


DEVELOP 

PMS  &  s>R8 

FUNCTIONIN'.. 

SURVEYS 


ADMINISTER 

SURVEYS 


INTEGRATED 
ANALYSIS 
OF  TPMS 
FUNCTIONAL 
COMPONENTS 


REPORT 


(7)  Provide  needed  support  and  assistance  to  help  study  team 
overcome  unforeseen  obstacles. 

b.  Division  Chiefs 

(1)  Participate  in  data  collection  including  individual  productivity 
assessments  of  branches. 

(2)  Provide  an  assessment  of  the  impact  of  TPMS  on  division 
managerial  practices,  productivity,  and  employee  morale. 

(3)  Afford  study  team  free  and  unrestrained  access  to  division 
personnel  for  the  required  data  collection  procedures. 

(4)  Encourage  full  cooperation  of  employees  with  study  effort. 

c.  Branch  Chiefs 

(1)  Participate  in  data  collection  including  individual  productivity 
assessment  of  branch  members. 

(2)  Afford  study  team  free  and  unrestricted  access  to  branch 
personnel  for  the  required  data  collection  procedures. 

(3)  Encourage  full  cooperation  of  employees  with  study  effort. 

d.  Study  Team 

(1)  Develop  project  plan  and  study  plan. 

(2)  Develop  study  instruments. 


lect  required  data. 


(4)  Conduct  study  in  accordance  with  study  plan  and  project  plan. 

(5)  Safeguard  sensitive  information  collected  as  part  of  this  study. 

(6)  Keep  Deputy  Director  of  Technical  Operations  informed  on  study 
progress,  including  a  briefing  of  Phase  I  results  by  23  December  1981. 

(7)  Prepare  a  report  of  Phase  I  results. 
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12.  SCHS 

DULt 

X 

Mi  1 estone 

Dc.  t  e 

a . 

Draft  Project  Plan 

Apr  a.i 

, 

b . 

Draft  Study  Plan 

1 

J  in 

$ 

c . 

TPASANA  technical  Staff  Survey 

/ 

1  ,  ’  ■ 

d. 

TPMS  :i-st  /ear  Results  Collected 

^  ■ 

] 

e . 

Criterion  Measure  Data  Collected 

il 

y. 

i 

f , 

Division  Chief  Interviews  Completed 

1U 

'*<.*■  11 

')■ 

Data  Analysis  Completed 

1 

Dec  HI 

h . 

Brief  Results  to  Deputy  Director  for 

Dec  Hi 

1 

Technical  Operations 

i . 

Phase  I  Report  Completed 

jy 

Jan 

s 

— 

s 

— 

i 

< 

— 

• 

• 
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TRASANA  TECHNICAL  STAFF  SURVEY 


B-I 


TRASANA  TECHNICAL  STAFF  SURVEY 


TURN  THE  PAGE  AND  READ  THE  INFORMATION 


AND  INSTRUCTIONS 


PURPOSE 


The  purpose  of  this  questionnaire  is  to  find  out  how  the  TRASANA 
Technical  Staff  views  the  TRASANA  Productivity  Measurement  System  (TPMS)  at 
this  time. 


GENERAL  INFORMATION 

This  questionnaire  has  been  designed  to  answer  specific  management 
questions  about  TPMS  as  a  part  of  the  evaluation  of  the  TPMS  three  year  trial . 
Honest  answers  to  the  questions  are  required  for  a  meaningful  evaluation.  You 
are  not  required  to  give  your  name,  but  your  organizational  unit  (office 
symbol)  is  needed  for  grouping  responses.  Results  will  be  presented  to 
management  in  the  form  of  summary  statistics  only. 

This  survey  has  three  parts.  Part  I  asks  general  background  information. 
Part  II  consists  of  several  attitude  statements  about  TRASANA  and  TPMS.  You 
are  to  indicate  on  the  answer  sheet  the  extent  to  which  you  agree  or  disagree 
with  each  statement.  You  will  probably  disagree  with  some  statements  and 
agree  with  others.  VJe  are  interested  in  the  extent  to  which  you  agree  or 
disagree  with  the  stated  opinions.  Part  III  has  three  questions  on  your 
familiarity  with  TPMS.  The  entire  survey  should  take  approximately  15 
mi nutes. 
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GENERAL  INSTRUCTIONS 


1.  Use  a  soft  lead  pencil  for  marking  the  answer  sheet. 

2.  Read  the  instructions  for  each  part  of  the  questionnaire. 

3.  Read  each  item  carefully,  and  be  sure  the  number  on  the  answer  sheet 
matches  the  item  you  are  answering. 

4.  Fill  in  the  circle  on  the  answer  sheet  which  3EST  describes  your  >-esDcrse 
to  the  statement.  Select  only  one  response  for  each  statement. 

5.  If  you  make  a  mistake  or  wish  to  change  your  answer,  erase  the  mark 
completely  before  entering  a  new  one. 

6.  Answer  every  item.  If  the  answers  provided  are  not  exactly  what  you  would 
like,  select  the  one  which  is  closest. 

ANSWER  SHEET 

1.  In  the  space  "NAME",  print  the  complete  office  symbol  for  your  branch.  If 
you  are  assigned  to  a  divisional  level  job,  enter  your  complete  office  symbol. 
EXAMPLE:  . 


NAME  (Last,  First.  M.l.) 

AT  A  A  -  TdIaI 

o  o  o  o  •  d  o‘  Q  O  Q  o  o  o  oo  o  o  o  o  o 

•  <2>##®®-®#®©®©®<5>®©®®®® 
®®®®'®'7fc®©®®,r  '5>®®®'^  ®®®® 

< >  S'- 


2.  Under  "SPECIAL  COOES"  enter  (left  justified)  the  four  number  designation 
for  your  career  specialty  area  if  you  are  a  civilian.  Military  personnel 
enter  duty  MOS.  EXAMPLE: 


3.  The  last  page  of  this  booklet  is  for  you  to  add  any  comments  about  TPMS  or 
the  three  year  test  of  TPMS  (TRASANA  MEMO  5-4)  which  you  feel  would  be  helpful 
to  the  test  evaluation,  including  suggestions  for  improvements. 
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PART  I:  BACKGROUND  INFORMATION 


INSTRUCTIONS 

Answer  all  questions.  Mark  your  answers 
on  the  answer  sheet  provided  with  a  soft 
lead  pencil.  Select  only  one  answer  for 
each  question. 


I.  I  am  a  (an) 

A.  Civilian 

B.  Active  duty  officer 

C.  Active  duty  enlisted  soldier 


2.  My  age  is 

A.  Under  30 

B.  30  thru  39 

C.  40  thru  49 
0.  50  thru  59 

E.  60  thru  64 

F.  65  thru  69 

G.  70  and  over 

3.  My  grade  Is  (If  not  civilian  mark  “I") 

A.  GS-07 

B.  GS-09 

C.  GS-1I 
0.  GS-12 

E.  GS-13 

F.  GS-14 

G.  GS-I5 

H.  Other 

I.  Not  a  civilian 

4.  My  military  pay  grade  Is  (If  not  active  duty  officer  mark  "G") 

A.  0-1 

B.  0-2 

C.  0-3 
0.  0-4 

E.  0-5 

F.  0-6 

G.  Not  active  duty  officer 


« 


X 


i 

1 


a 


5.  My  military  pay  grade  is  (It  not  active  duty  enlisted  soldier  mark  "G") 

A.  E-4 

B.  E-5 

C.  E-6 
3.  E-7 

E.  E-3 

F.  E-9 

G.  *iot  active  duty  enlisted  soldier 

6.  I  have  been  in  my  present  grade  for 

A.  Less  than  I  year 
3-  1  year  but  not  2 

C.  2  years  but  not  3 
0.  3  years  but  not  4 

E.  4  years  but  not  5 

F.  5  years  but  not  6 

G.  S  years  but  not  7 

H.  7  years  but  not  8 

I.  8  years  but  not  9 

J.  9  years  or  more 

7.  I  have  worked  for  TRASANA  (or  its  SAFEGUARD  predecessor)  for 

A.  Less  than  1  year 

B.  I  year  but  not  2 

C.  2  years  but  not  3 

D.  3  years  but  not  4 

E.  4  years  but  not  5 

F.  5  years  but  not  6 

G.  6  years  but  not  7 

H.  7  years  but  not  8  (SAFEGUARD) 

I.  8  years  but  not  9  (SAFEGUARD) 

J.  9  years  or  more  (SAFEGUARD) 

8.  I  have  worked  for  the  Federal  Government  (including  military  service)  for 
a  total  of 


A.  Less  than  1  year 
8.  1  thru  3  years 

C.  4  thru  9  years 

D.  10  thru  19  years 

E.  20  thru  29  years 

F.  30  years  or  more 

9.  The  highest  academic  degree  or  diploma  I  have  is 

A.  High  School  Oiploma/GED 
8.  Associate  degree 
C.  Bachelor's  degree 
0.  Master's  degree 
E.  Ooctor's  degree 
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10.  My  primary  academic  specialty,  or  the  area  in  which  I  have  had  the 
highest  level  of  advanced  study,  is 

A.  Engineering 

B.  Physical  sciences  (physics,  chemistry,  etc.) 

C.  Mathematics/statistics 
0.  Computer  science 

E.  Social  sciences  (psychology,  sociology,  etc.) 

F.  Education 

G.  Business  (management,  accounting,  etc.) 

H.  Liberal  arts  (music,  art,  history,  etc.) 

I .  Other 

11.  My  position  is  best  described  as 

A.  Clerical 

B.  Technical /Professional  (team  chief,  team  member,  support) 

C.  Advisory  ( non-supervi  sory ) 

D.  First-line  supervisor  (Branch  Chief) 

E.  Second-line  supervisor  (Division  Chief  or  Deputy) 

F.  None  of  the  above 

12.  I  am  a 

A.  Male 

B.  Female 

13.  My  primary  ethnic  identity  is 

A.  American  Indian 

B.  Hispanic 

C.  Afro-American 
0.  Asian  American 
E.  Other 


GO  ON  TO  PART  II 
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PART  I r :  TPMS  OPINION  Sl'RVEv 


INSTRUCTIONS 

This  section  of  the  survey  contains  a  series  of  attitude  statements 
designed  to  determine  how  the  staff  feels  about  TPMS  and  its  relation  to  the 
type  of  work  we  do  at  TRASANA.  Read  each  statanent,  and  decide  wnether  you 
agree  or  disagree  with  it.  Next,  select  the  letter  that  BEST  describes  to 
what  extent  you  agree  or  disagree  with  the  statement,  and  mark  the  circle 
under  the  letter  on  the  answer  sheet.  In  the  event  that  none  of  the  six 
responses  adequately  reflects  your  opinion  on  the  statement,  select  the  one 
which  is  closest  to  the  way  you  feel,  or  which  is  least  objectionable  to  you. 
Answer  all  questions. 


CHECK  TO  3E  SURE  YOU  ARE  AT  THE  RIGHT 
PLACE  OH  THE  ANSWER  SHEET. 


Agree  j  Disagree  ^ 


14.  Overall,  the  performance  standards  at 
TRASANA  are  high. 

15.  I  am  proud  to  pe  associated  wi  th 
TRASANA  oecause  of  its  quality 
products. 

16.  I  like  worki ng  at  TRASANA. 

17.  There  is  a  Tot  of  "dead  wood*  at 
TRASANA. 

18.  Most  of  the  people  at  TRASANA  do  high 
quality  work. 

19.  The  management  at  TRASANA  does  its 
best  to  provide  good  working 
conditions. 

20.  Supervisors  in  TRASANA  will  take  care 
of  their  people,  with  or  without  TPMS. 


8E  SURE  YOU  ARE  AT  21  ON  THE  ANSWER  SHEET 


A 

A 


SCO 
3  C  0 

SCO 


E  F 

E  F 


21.  Most  people  are  treated  fairly  at 
TRASANA. 


A  3  C  0  E  F 


22.  The  overall  productivity  of  TRASANA 
is  not  really  very  high. 

2  .  If  TRASANA  were  on  a  profit  basis, 
such  as  a  private  contractor  or 
consulting  firm,  it  would  not  be 
making  money. 

24.  Productivity  in  my  branch  often 
suffers  from  inadequate  organization 
and  planning. 

25.  Productivity  in  my  division  often 
suffers  from  inadequate  organization 
and  planning. 


A 

A 


A 


A 


26.  I  like  doing  the  kind  of  work  I  do  at 
TRASANA. 


A 


SCO 

SCO 

BCD 

SCO 

SCO 


E  F 

E  F 


E  F 


E  F 


E  F 
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Somewhat 


ts 


r  —  - 

I 

I 

!  * 

ii 


Agree 


r 


Qlsagree 


1 


27.  '•y  job  is  not  very  -haP  engi  ng. 

23.  t  am  general1/  satisfied  with  my  joo 
performance. 

29.  I  do  not  get  much  personal 
satisfaction  --om  my  ;oo. 

30.  Tv’e  skll's  I  nave  are  «eP  matched  to 
the  requirements  of  my  j0t>. 


3E  SURE  /OU  ARE  AT  31  ?N  ’HE  ANSWER  SHEE’  ; 

i 

I 

31.  !  1o  not  get  much  useful  feedback  on  j 

how  1  am  doing  i  n  ,my  job.  | 

I 

32.  *y  supervisors  ganeraPy  give  me  the  i 

respect  I  deserve.  j 

33.  My  supervisors  do  not  know  what  I  can  j 

do.  | 

1 

34.  My  supervisor  does  a  good  Job  of  i 

planninq  work  and  managing  people.  ! 

I 

35.  I  am  usually  free  to  decide  on  my  own  i 

how  to  go  about  accomplishing  the 
tasks  I  am  given  to  do.  I 

36.  I  usually  have  very  little  say  in  the  I 

schedule  for  completing  my  tasks.  ! 

i 

37.  I  can  use  a  lot  of  initiative  and  try  i 

new  things  In  my  job.  I 

I 

38.  My  job  typically  Involves  doing  a  I 

small  part  of  a  large  project  for  i 

which  someone  else  <s  reponslble.  I 

39.  Quantifying  oroductlvlty  In  the  kind  I 

of  work  I  do  can  be  done  wl  th  rea-  I 
sonable  accuracy  (l.e.  distinguishing  | 

between  the  quality  of  contribution  I 

made  by  different  people  over  a  oerlodi 
of  time) .  I 


x  — 

V  3 

i  z 

(✓>  v» 


3 

3 

3 

3 

3 

3 

3 

3 


j 

I 


c  a 

c  3 


C  3  E 

C  D  E 


F  i 
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Somewhat 


40.  TPASAHA  productivity  can  be  Increased  A  3  C  3  E  F 
substantially  without  hiring  more  I 
peopl  e.  I 


41.  *ost  people  are  giving  TPMS  a  fair  i  A  3  C  3  E  F| 
chance.  i  ; 


42.  TPMS  is  a  simple,  easy  to  understand  !  4  3  C  U  E 

systan.  ! 

I 

43.  TPMS  is  flexible  enough  to  accontmodatel  A  3  C  D  E 

most  of  the  projects  done  in  TPASANA.  1 

I 

44.  TPMS  places  emphasis  on  the  factors  I  A  3  Z  3  E 

that  reflect  differences  in  I 

productivity.  I 

! 

45.  The  distribution  of  productivity  I  A  3  C  3  E 

points  will  show  how  the  study  1 

manager  views  the  work  quality  of  I 

individual  team  members.  I 

I 

46.  Productivity  points  and  indices  do  1  A  3  C  0  E 

not  really  matter  right  now  because  I 

the  system  Is  just  being  tested.  I 

I 

47.  With  experience,  manpower  resources  I  A  3  C  D  E 

required  for  projects  can  be  ! 

real Istlcally  estimated.  I 

I 

48.  with  experience,  computer  resources  i  A  3  C  D  E 

required  for  projects  can  be  ! 

realistically  estimated.  I 

1 

49.  I  have  seen  some  positive  effects  of  I  A  3  C  0  E 

TPMS  so  far.  I 

I 

50.  In  my  division,  most  people  have  I  A  3  C  0  E 

made  up  their  minds  about  TPMS.  I 

I 

51.  I  do  not  expect  to  score  very  well  1  A  3  C  0  E 

under  TPMS.  I 

I 

52.  Considering  that  measuring  I  A  3  C  0  E 

productivity  is  not  easy,  TPMS  is  I 

basically  a  fair  system.  I 

I 

53.  In  the  long  run,  all  divisions  will  I  A  3  C  0  E 

have  an  equal  chance  to  earn  I 

productivity  points  under  TPMS.  I 
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I  Agree 


I 


DU 


‘^“S  U  oojective.  A 

-'“odI  e  *no  produce  the  ■>:st  wo'x  ' 

wi '.  ’  aeog'-jMy  :et  the  nest 
productivity  points  .nder'  TMS- 

’~:MS  :arr  oe  easily  oan'cu  ited.  A 

I  understand  the  mechanics  ~f  "?MS.  \ 

l 

vast  of  the  technical  stacf  do  ’of  a 

understand  T>*.-S.  i 

i 

I  know  the  five  factors  used  -'o  '  A 

calculating  a  project  score.  1 

I 

t 

I  do  not  know  how  a  project  scare  is  !  A 
computed.  i 

I 

-JRM  ANSWER  SHEET  OVER  t 

I 

I 

I  understand  the  difference  between  !  A 

productivity  points  (PP)  and 
productivity  index  (PI).  I 

t 

Given  all  the  Input  factor  values,  I  I  A 

cpuld  compute  my  own  productivity 
index  right  now.  I 

I 

As  Tong  as  a  person  does  his  Job,  he  I  A 

need  not  worry  about  TPMS  because  he  I 
will  come  out  okay.  I 

I 

For  the  most  part,  productivity  oolnesi 
are  distributed  according  to  the  i  A 

amount  of  time  spent  on  a  project.  I 

I 

If  a  division  chief  feels  an  I  A 

Individual  has  not  been  justly 
rewarded  for  work  on  a  project,  he  I 

can  create  extra  productivity  points  i 

for  that  person.  I 


3 


3 

3 

3 

3 

3 


3 


3 


3 


3 

3 


D 


C  3 

C  0 

C  D 

:  d 

C  3 
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66.  A  person  has  the  option  of  not  using 
time  charged  to  overhead  activities 
when  calculating  Ms  productivity 
1 ndex . 

57.  Jnder  TPMS,  a  study  manager  is 

encouraged  to  worn  for  the  highest 
quality  (PR8)  score,  even  at  the 
expense  of  being  late  or  over  an 
manpower. 

53.  Productivity  points  and  .merit  pay 
are  not  related  at  this  time. 

69.  TPMS  will  only  affect  management. 

70.  Management  practices  have  been 
changed  by  TPMS. 


BE  SURE  YOU  ARE  AT  71  ON  THE  ANSWER  SHEET 


71.  More  planning  is  being  done  for 
projects  now  than  before  TPMS. 

72.  TPMS  will  be  a  useful  tool  for 
management. 

73.  After  the  bugs  are  worked  out,  TPMS 
will  be  a  useful  factor  for 
Individual  performance  appraisals. 

7A.  My  productivity  index  (PI)  under 
TPMS  is  pretty  much  beyond  my 
control . 

75.  My  supervisor  has  discussed  with  me 
the  distribution  of  productivity 
points  on  the  projects  on  which  I 
have  worked. 

76.  Management  practices  have  Improved 
as  a  result  of  TPMS. 

77.  TPMS  represents  no  major  change  In 
emphasis  by  TRASANA  management. 


Agree 

l 

Disagree 

1 

_ 1 

.  >* 

! 

>»  >* 

1 

-  f 

a* 

1  e 

i 

Z  1  z 

-c  .  r: 

2*  j 
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s 

1 

1 

u  i 

l  ^ 

1 
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3 
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79.  "3ms  *ill  motivate  peco' ■?  to  *cHc  4  3'.; 

larger.  | 

1 

79.  My  serrorjiarce  aooraisal  *or  this  4  3  3  D 

/ear  will  not  ae  influenced,  ay  *C,*S. 

30.  'hHS  «ill  not  affect  ne  /ery  -men.  '4  3  3  3 

I 

91.  Assuming  that  3roduct4<ity  tan  he  4  3  C  3 

increased  by  doing  more  «ork  or  ; 

doing  higher  quality  <ori,  '?MS  «’ 1 1  i 
lead  to  increased  productivity  in  | 

the  long  run.  | 

I 

32.  TPMS  has  -esutted  in  a  more  |  A  3  C  3 

camoetitive  environment  among  the  I 

7RASANA  staff.  | 


33. 

If  TPMS  is  adopted  at  TRASANA ,  I 

things  will  not  really  change  very 
much.  | 

r 

A 

3 

C 

3 

E 

F 

34. 

l 

TPMS  has  substantially  affected  my  | 

branch.  j 

l 

A 

3 

c 

0 

E 

F 

35. 

1 

If  TPMS  is  adopted  at  TRASANA,  1 

productivity  will  not  change.  1 

| 

A 

B 

c 

D 

E 

F 

36. 

I 

I  do  not  care  whether  TPMS  succeeds  or| 
fails  in  its  three  year  test.  1 

A 

3 

c 

0 

E 

c 

37. 

1 

I  do  not  know  very  much  about  TPMS.  | 

| 

A 

3 

c 

D 

E 

F 

88. 

1 

I  would  like  to  know  more  about  TPMS.  | 

A 

3 

c 

0 

E 

F 

GO  ON  TO  PART  III 
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j  Somewhat 


PART  III:  TPMS  FAMILIARITY  AND  COMMENTS 


INSTRUCTIONS  I 

I 

Thfs  section  of  the  survey  contains  questions  1 
designed  to  find  out  how  familiar  the  TRASANA  I 
staff  is  with  various  aspects  of  TPMS  at  this  I 
time.  Read  each  question  carefully  and  record  I 
your  answers  on  the  answer  sheet.  Answer  all  ! 
questions  I 

I 


CHECK  YOUR  ANSWER  SHEET  TO  3E  SURE  YOU  ARE  ON  THE  CORRECT  NUMBER 


89.  Which  of  the  following  best  describes  your  familiarity  with  TRASANA 
Memorandum  5-4? 

A.  I  do  not  know  what  it  is. 

B.  I  know  what  it  is  but  have  not  seen  it. 

C.  I  have  seen  it,  but  have  not  looked  through  it. 

D.  I  have  browsed  through  it  quickly. 

E.  I  have  read  it. 

F.  I  have  read  it  more  than  once. 

G.  I  have  studied  it  thoroughly. 

90.  Which  of  the  following  best  describes  your  overall  understanding  of  TPMS 
at  this  time? 

A.  I  fully  understand  it. 

B.  I  understand  It  fairly  well. 

C.  I  do  not  understand  some  parts  of  it. 

D.  I  understand  very  little  of  it. 

E.  I  do  not  know  anything  about  it. 

91.  Which  of  the  following  statements  best  describes  your  overall  feeling 
about  TPMS  at  this  time? 

A.  I  like  it. 

B.  There  are  some  good  and  bad  points  to  it,  but  the  good  probably 
outweigh  the  bad. 

C.  There  are  some  good  and  bad  points  to  it,  but  the  bad  probably 
outweigh  the  good. 

0.  I  don't  like  it. 


Please  use  the  attached  page  for  comments  about  TPMS  which  would  be  helpful  to 
the  evaluation  team,  including  suggestions  for  changing  TPMS  or  5-4.  Thank 
you. 


APPENDIX  C 


SUPPORTING  TABLES 


TABLE  C-l 


TTS  AND  SURVEY  SAMPLE  BY  DIVISION 


DIVISION 

PERCENT  OF  SAMPLE 

A 

12.6 

11.1 

l  B 

9.5 

10.5 

i 

C 

13.0 

12.6 

0 

1 

15.0 

15.  P 

E 

14.6 

17.9 

F 

11.1 

7.4 

G 

3.3 

6.3 

H 

15.8 

18.4 

TOTAL 

99.92 

100 

aDoes  not  total  100 %  due  to  rounding. 
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TABLE  C-2 

AGE  OF  RESPONDENTS 


mMMUMwxm 

Under  30 

21 

11.1 

30  to  39 

57 

30.0 

40  tc  49 

78 

41.1 

50  to  59 

29 

15.3 

60  to  64 

4 

2.1 

65  to  69 

1 

0.5 

TOTAL 

190 

ioo. \a 

aDoes  not  total  100 %  due  to  rounding. 


TABLE  C-3 

RESPONDENT  TIME  AT  TRASANA 


PERCENT 

Less  than  1  yr 

1 

21 

2 

8 

■9 

3 

3 

1.6 

4 

7 

3.7 

5 

11 

5.8 

6 

19 

10.0 

7 

0 

0 

8 

6 

3.2 

9  or  more 

90 

47.4 

TOTAL 

190 

a 

100.1 

aDoee  not  total  100 %  due  to  rounding. 
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TABLE  C-4 

RESPONDENT  TIME  IN  GRADE 


TIME  (YEARS) 

NUMBER 

PERCENT 

Less  than  1  yr 

31 

16.3 

1 

17 

8.9 

2 

12 

6.3  j 

3 

16 

8.4  j 

4 

16 

8.4 

5 

20 

10.5 

6 

5.3 

7 

8 

4.2 

8 

6 

3.2 

9  or  more 

54 

28.4 

TOTAL 

190 

a 

99.9 

aDoee  not  total 

rounding. 

TABLE  C-5 

RESPONDENT  TIME  IN  FEDERAL  SERVICE 


Less  than  1  yr 

3 

1.6 

1  to  3 

13 

6.8 

4  to  9 

36 

18.9 

10  to  19 

75 

39.5 

20  to  29 

60 

31.6 

30  or  more 

3 

1.6 

TOTAL 

190 

100 

TABLE  C-6 


HIGHEST  ACADEMIC  DEGREE  OF  RESPONDENTS 


High  School/GED 

8 

4.2 

Associate 

4 

2.1 

Bachelor 

91 

47.9 

Master 

79 

41.6 

Doctor 

8 

4.2 

TOTAL 

190 

100 

TABLE  C-7 

PRIMARY  ACADEMIC  SPECIALTY  OF  RESPONDENTS 


mmmm 

Mathematics/Statistics 

56 

29.5 

Engi neering 

54 

28.4 

Physical  Sciences 

32 

16.8 

Computer  Science 

12 

6.3 

Social  Sciences 

11 

5.8 

Education 

9 

4.7 

Business 

7 

3.7 

Liberal  Arts 

3 

1.6 

Other 

6 

3.2 

TOTAL 


190 


100 


TABLE  C-8  (CONTINUED) 


mu - 

n  DISAGREE 

S 

S 

S 

s 

S 

S 

T 

0 

L 

L 

0 

T 

ITEM 

R 

M 

1 

I 

M 

R 

0 

E 

G 

G 

E 

0 

NUMBER 

N 

W 

H 

H 

W 

N 

G 

H 

T 

T 

H 

G 

L 

A 

L 

L 

A 

L 

Y 

_ T 

Y 

Y 

T 

Y 

A 

B 

C 

HI 

E 

F 

31 

15 

21 

18 

12 

19 

15 

32 

39 

33 

14 

5 

3 

6 

33 

7 

12 

17 

11 

30 

23 

34 

23 

36 

19 

7 

9 

6 

35 

51 

27 

14 

4 

2 

2 

36 

12 

11 

12 

18 

24 

23 

37 

39 

31 

14 

7 

5 

4 

38 

19 

19 

12 

8 

17 

25 

39 

5 

14 

17 

11 

22 

31 

40 

22 

29 

25 

6 

13 

5 

41 

10 

33 

38 

7 

8 

4 

42 

3 

15 

15 

18 

22 

27 

43 

4 

21 

18 

20 

18 

20 

44 

1 

13 

26 

20 

14 

26 

45 

7 

21 

23 

13 

17 

19 

46 

4 

11 

16 

21 

19 

29 

47 

10 

39 

20 

12 

9 

10 

48 

8 

30 

23 

16 

12 

11 

49 

4 

12 

18 

18 

18 

30 
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TABLE  C-8  (CONTINUED) 


TABLE  C-8  (CONTINUED) 


TABLE  C-9 


TECHNICAL  STAFF  SURVEY  RESPONSES  TO 
ITEMS  89  TO  91 
(PERCENTAGES) 


ITEM 

RESPONSE 

0/ 

A. 

I  do  not  know  what  it  is. 

11 

j 

) 

B. 

I  know  what  it  is  but  have  not  se^n  it. 

?  i 

C. 

I  have  seen  it,  but  have  not  looked  through  it. 

i  ! 

89 

D. 

I  have  browsed  through  it  quickly. 

24 

E. 

I  have  read  it. 

9? 

F. 

I  have  read  it  more  than  once. 

30  ; 

l 

G. 

I  have  studied  it  thoroughly. 

9  i 

i 

A. 

I  fill  ly  understand  it. 

! 

! 

5  i 

B. 

I  understand  it  fairly  well. 

j 

42 

90 

c. 

I  do  not  understand  some  parts  of  it. 

n 

I  understand  very  little  of  it. 

B 

E. 

I  do  not  know  anything  about  it. 

3  I 

A. 

I  like  it. 

1 

91 

B. 

There  are  some  good  and  bad  points  to  it,  but 
the  good  probably  outweigh  the  bad. 

33 

C. 

There  are  some  good  and  bad  points  to  it,  but 
the  bad  probably  outweigh  the  good. 

47 

D. 

I  don't  like  it. 

19 
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TABLE  C-10 


COMPARISON  OF  SUPERVISOR  AND 
NON-SUPERVISOR  SURVEY  RESPONSES 


CLUSTER 

wmimfsvmsm 

IhHBm 

t-VALUEa 

TRASANA 

4.92h 

4.19 

4.41 

(.76) 

(.80) 

Job  Satisfaction 

4.84 

4.43 

2.30 

(.84) 

(.89) 

Control  of  Work 

5.06 

4.52 

2.62 

(.93) 

(1.09) 

General  Reaction 

3.84 

3.36 

2.90 

(.76) 

(.74) 

Fairness 

3.56 

2.82 

2.98 

(1.15) 

(1.18) 

Understand  ng 

4.85 

3.84 

6.85 

(.66) 

(.79) 

Management  Tool 

3.92 

3.35 

4.06 

(.64) 

(.70) 

TOTAL 

4.29 

3.71 

6.52 

(.41) 

(.44) 

aA ll  t-values  are  significant  beyond  the  .05  level  ( critical  values: 
p  <  .05  =  1.96 ;  p  <  .01  =  2.58;  p  <  .001  =  3.29). 

b Top  number  is  the  mean  and  the  number  in  parenthesis  is  the  standard 
deviation. 
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aDoes  not  total  100%  due  to  rounding. 


[ABLE  C-12.  TPMS  SCORING  DATA  BY  BRANCH 


BRANCH 

PROJECTS1 

TMM 

REGULAR 

MAN  DIR 

JUDO 

DISC 

DISC 

TOIAL 

PI 

OVERHEAD 

OVERHEAD 

AWARDS4' 

AWARDS 

PP 

PP 

1 

2 

58.41 

4.06 

0.43 

0 

0 

0 

108.80 

1.81 

2 

1 

80.7/ 

6.75 

0. 11 

0 

0 

0 

136.73 

1.64 

3 

10 

44.33 

3.02 

4.43 

-2 

2 

2.36 

72.48 

1.63 

4 

3 

31.17 

0.50 

0.45 

0 

0 

0 

50.92 

1.63 

5 

8 

61.99 

7.80 

1.01 

*1 

0 

0 

105.09 

1.62 

6 

3 

165.29 

2.71 

0.23 

0 

0 

0 

265.28 

1.59 

7 

10 

106.26 

2.68 

0.07 

-1.  *1 

1 

1.50 

168.94 

1.58 

8 

4 

35.00 

7.35 

0.57 

0 

0 

0 

58.74 

1.58 

9 

7 

73.36 

9.44 

0.54 

0 

0 

0 

121.14 

1.57 

10 

12 

36.18 

5.70 

0.90 

0 

1 

1.0 

58.88 

1.55 

11 

9 

98.09 

16.63 

0.38 

0 

5 

2.08 

165.31 

1.52 

12 

9 

32.64 

16.72 

0.16 

0 

3 

0.75 

57.51 

1.50 

13 

2 

22.74 

13.72 

0.85 

0 

0 

0 

39.53 

1.47 

14 

10 

28.89 

9.01 

0.18 

0 

1 

0.25 

45.36 

1.44 

lb 

12 

54.74 

33.07 

0 

0 

2 

2.45 

91.59 

1.42 

16 

3 

3.68 

4.03 

0. 1 

0 

0 

0 

6.87 

1.42 

17 

4 

32.43 

4.55 

2.69 

0 

0 

0 

45.98 

1.41 

18 

2 

19.86 

3.56 

0 

0 

0 

0 

28.76 

1.38 

19 

4 

33.46 

44.14 

4.73 

0 

1 

0./6 

57.63 

1.3b 

20 

4 

16.21 

11.44 

1.94 

0 

2 

4.00 

19.98 

1.33 

21 

17 

21.78 

50.34 

0 

0 

3 

8.0 

43.77 

1.30 

22 

8 

28.46 

12.70 

0.11 

0 

0 

0 

39.49 

1.27 

23 

2 

17.57 

13.12 

0.56 

0 

0 

0 

24.27 

1.23 

24 

2 

0.90 

4.96 

0.35 

0 

0 

0 

1.59 

1.17 

NOTE:  These  data  were  taken  from  the  TPMS  scoring  sheets  submitted  by  each  division  office,  branches 
are  arranged  from  highest  to  lamest  PI. 


aIncludee  the  total  number  of  projects  for  which  productivity  points  mere  awarded  to  the  branch. 

^ Only  interbranch  judgmental  awards  (JA)  are  shown.  A  negative  JA  means  that  the  branch  received  up  to  JO 
percent  less  PP's  than  earned  based  solely  on  manpower  charges.  A  positive  JA  means  the  branch  received 
up  to  JO  percent  more  PP's  than  earned  based  solely  on  manpower  chargee.  All  of  these  JA'6  were  in  the 
same  division. 
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IMM 

. . . . J 

"k'.OLAk ' 
UVtPHlAI) 

MAN  OIK 
UVt  RH  A 

280.4 

1  1. 

(4) 

1.31 

2  1  3.98 

78.55 

(1?) 

5.64 

7  7.66 

77.39 

(35) 

1.81 

158.38 

11.10 

(7) 

1.38 

159.94 

66.  74 

(42) 

0.87 

87.  ID 

78.67 

(90) 

1  '5 

110.01 

108.16 

m 

5.80 

48.59 

31.57 

(65) 

2.56 

IN:  DATA 

Uv  HI J 13 ION 

1006  ‘ 

1  *  ‘  ~ 

DIM. 

^  0  1  S'. 

|  'O’.Ai 

SWAkI.^ 

AWARDS 

PP 

K*' 

[ - 

0 

0 

i 

0 

1 

-i 

1 

1 

3. 8(> 

1  349.21 

0 

0 

0 

1  if .  08 

0 

1 

0.4 

251.07 

0 

6 

2.85 

|  263.31 

*1 

3 

9.25 

152.00 

0 

2 

2.45 

137.81 

0 

2 

, 

4.00 

72.83 

NOTE:  These  data  we re  taken  from  the  TPMS  scoring  sheets  submitted  by  each  division  office.  Divisions  are 
onic-ed  by  PI. 

aIncludes  only  projects  for  which  the  division  was  primarily  responsible, 
b Number  in  parentheses  are  percent  of  TMM,  rounded  to  nearest  whole  number. 

°Only  interdivision  JA  's  are  shown.  One  division  received  lees  PP's  and  another  received  more  than  earned  based 
solely  on  manpower  charges  for  one  project. 
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Productivity  Index  •  jjl 

bum  of  column  ?  4 

1.  Second  overhead  charge  accounts  as  separate  projects. 

2.  Entry  In  column  3  is  PS  x  >IM. 

3.  Entries  marked  with  an  asterisk  are  optional. 
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Productivity  Index 


Sum  of  column  #11 
Sum  of  column  #  4 


1.  Record  overhead  charge  accounts  as  separate  projects 

2.  Entries  marked  with  an  asterisk  are  optional. 


1.  Record  overhead  charge  accounts  as  separate  projects. 

2.  Entries  marked  with  an  asterisk  are  optional. 
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APPENDIX  E 


MATHEMATICAL  ANALYSIS  OF 
THE  TPMS  SCOP  I  NO  ALGORITHM 


1.  GENERAL 

This  appendix  provides  the  detailed  mathematical  analysis  of  the  TPMS  scorinn 
a  1 oori thm. 

?.  the  Td’LS  MEASUREMENT  FMf'OTf ON 

The  assignment  of  the  productivity  index  (pI)  hv  the  measurement  function  can 
he  conveniently  seomented  into  a  sequence  of  assignments  bv  component  func¬ 
tions.  first,  a  project  scorinn  function  yields  a  nro.iect  score  (S)  from 
relevant,  variables.  Next,  us  inn  the  concent,  of  productivity  points  (pP),  this 
score  is  apportioned  amonq  the  units  and  individuals  who  contributed  to  the 
project,  in  relation  to  manpower  expended  and  to  the  quality  of  the  contribu¬ 
tion.  Finally,  the  PI  for  the  individual  or  the  unit  is  given  as  an  average 
of  project  PP  earned,  weighted  by  the  manpower  expended  on  the  projects. 

a.  Project  Scoring  Function 

The  S  is  the  product  of  five  project  variables:  manpower  utilization 
efficiency  (R),  computer  utilization  efficiency  (C),  timeliness  (T),  quality 
(n),  and  priority  (P).  Thus 


S  =  RxCxTxOxP 


Each  of  these  five  factors  is  discussed  separately. 

(1)  Priority.  Originally  the  value  of  P  was  to  vary  from  1.0  to  1.? 
with  the  value  of  1.?  assiqned  to  external  projects.  For  internal  projects 
the  value  of  P  was  to  he  determined  bv  the  PMB.  Since  all  projects  were 
assiqned  a  P  value  of  1.2  durinq  the  first  scorinq  period,  this  variable  will 
be  held  constant  in  the  subsequent  analysis. 

(?)  Oualitv.  The  value  of  0  is  assiqned  by  the  PRR  upon  review  of 
the  project  final  report.  Values  of  0  may  vary  from  1.00  to  1.50. 

(3)  Manpower  Utilization  Efficiency.  The  value  of  R  is  determined 
from  the  planned  manpower  usaqe  for  the  project  (Rp)  and  the  actual  manpower 
used  on  the  project  (Ra)  as  follows: 


R  = 


f 0.7  when  Rp/RA  <  0.5 
1.0  when  Rp /Ra  =1.0 
1.2  when  Rp/RA  > 


Intermediate  values  of  R  scale  up  linearly  between  1.0  and  1.2  and  scale  down 
linearly  between  1.0  and  0.7.  The  expression  for  R  as  a  piecewise  linear 
function  of  Rp/Ra  is  then 


0.7  when  Rp/RA  <0.5 

0.6  (Rp/RA^  +  0.4  when  0,5  Rp/RA  <  1.0 

0.2  (Rp/Ra)  +  0.8  when  1.0  _<  Rp/RA  <  2.0 

1.2  when  2.0  Rp/Ra 


The  qraph  of  this  function  is  shown  in  fiqure  E-l. 


Fiqure  E-l.  Manpower  Utilization  Efficiency  (R) 


(4)  Computer  utilization  efficiency.  The  value  of  f  is  determi ned 
from  the  planned  computer  usage  for  the  project  (Cp)  and  fhp  actual  computer 
usane  { r/\ )  as  follows: 

f  O.o  when  f  r.  ,r^  '  o.t, 
i 

r  -  ^1.0  when  rp/fA  -  l.D 

[  1.1  when  '  wr,  p.n 

Intermediate  va1ues  of  r  <-,ra!e  up  and  ■‘own  linpr.irlv  f ro’i  r  =  1.0.  The 
expression  for  r  as  a  r>  i  e^ew  i  sp  linear  ?un''*’on  of  rp/r^  vs: 

r>  t  r  j\  <  0  #  c 

Tp  ,  '  n  -  ■  -  #  ’  whf»"  1  ’ .  ^  f.  p  /C/\  f'  ]  . 

.  “  w 1 .  "•  £  r  D  ,-r <  /  .  o 

when  ”  <  rp  /f  ^ 


Figure  F-?  is  the  graph  of  this  f  * --.r .  "  r  mo  +tie  course  of  the  first 

scorinq  period  the  neasurp  of  (o->p  ,».*--  ow  was  -Ranged.  because  of  this,  hi 
of  the  61  proiect.s  scored  in  fhp  ipn  e  wer«  administratively  assinned  a 

C  value  of  1. 

(5)  Timeliness.  The  value  of  1  is  a  function  of  the  planned 
completion  date  and  the  actual  completion  daf»>.  The  value  may  vary  between  a 
maximum  value  of  1.1  for  early  completion  and  a  minimum  value  of  O.R  for  late 
completion.  The  functional  relationship  vieldinq  T  is  project  d^oendent  and 
is  determined  by  the  project  study  manaoer  with  the  approval  of  the  PMB. 

Fiqure  F-3  qives  the  timeliness  value  as  a  function  of  time  for  one  project. 

(6)  Project  score  values.  Values  for  project  scores  can  vary  from 
0.604  to  2.6136.2  A  value  of  1.56  miqht  be  viewed  as  a  "base"  score  - 
representinq  a  project  with  minimally  acceptable  quality  (R  or  0  =  1.3), 
exactly  on  time  (T  =  1.0),  and  usinq  exactly  planned  manpower  (R  =  1.0)  and 
computer  usaqes  (C  =  1.0).  The  minimum  score  for  a  project  of  minimal  \y 
acceptable  quality  is  0.7R62.  Table  F-l  shows  maximum,  minimum,  "base",  and 
minimally  acceptable  project  scores  with  0  varying  in  the  first  case,  and  held 
constant  at  1.0  in  the  second  case.  The  project  score  has  a  wider  range  of 
possible  values  when  C  is  not  held  constant. 


2All  projects  scored.  in  the  first  period  have  P  =  2.2.  When  calculated  with, 
aid,  variables  eaual  to  the  loudest  value  except  for  P  =  2.2,  the  lowest  score 
is  n.  PQ4P. 


!  O.Q 


e  i 


1.1 


Flqure  E-3.  Timeliness  (T) 
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PPOJFrr  SCORES  FOP  PFDRF SFMTAT! VF  IMPi'TS 
Mariablp  Value  of  C 


Case 
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r 

T 

c 

Ma.x  imum 
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Min; 
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7 

0.  0 

o. 

n,  u  ',;  ,■ 

°  a  s  p 

l.? 

1.3 

l.o 

1.0 

1.0 

1 .  SO 

Mi p i ha i i v 

Ac  c  p  o  t  a  b  1  p 

1.? 

1.3 

o.7 

o.o 

n  p 

0,. 

Constant 

value  of 

r 

Case 

P 

R 

c 

T 

Ma x imum 

l.P 

1.6 

1.? 

1 

l 

1.1 

?.  30? 

Mi n imum 

l.P 

1.0 

0.7 

1 

0.8 

0.67? 

Rase 

1.? 

1.3 

1.0 

1 

1.0 

1.66 

Minimally 

Acceptable 

1.? 

1.3 

0.7 

1 

0.8 

0.8736 

h.  Productivity  Point  Oistribution 

Productivity  points  are  qenerated  for  a  project  by  multiplyinq  the  project 
score  by  the  actual  manpower  usaqe  (in  man-months) . ? 


PP  =  S  x  Rfl 


These  PP  are  then  distributed  amonq  the  units  and  individuals  who  charqed  time 
to  the  project.  Basically  an  individual  or  unit  is  awarded  PP  in  relation  to 
the  manpower  charqed  to  the  project.  However,  the  quality  of  an  individual’s 
or  unit's  contribution  can  result  in  a  plus  or  minus  30  percent  deviation  from 


^'Overhead  nan  he  treated  as  a  orojeet  with  a  snore  of  1  for  general  overhead, 
or  2,  for  management  directed  overhead. 
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this  straiqht  charqe  to  manpower  award.  In  addition,  the  distribution  of  PP 
is  done  in  a  hierarchical  manner  which  makes  deviations  beyond  30  percent 
possible  in  some  cases.  Points  are  distributed  first  to  divisions,  then  to 
branches  within  a  division,  and  finally  to  individuals  within  a  branch.  At 
anv  of  the  3  levels  ir.  the  distribution  a  plus  or  minus  30  percent  deviation 
from  a  straiqht  charqe  to  manpower  award  is  possible. 

(1)  Oivision.  If  division  i  charqes  M-j  man-months  to  a  pro.iect  it 

earns 


PP  x  (Mi;QA)  x  di 
or 

S  x  Mj  x  d-j  productivity  points 
where  0.7  <_  d-j  £  1.3 


The  d-j  is  the  division  .iudqmental  factor.  If  the  award  is  strictly  accordinq 
to  manpower  charqed,  d-j  =  1.  Since  all  manpower  charned  orininates  from  some 
division  and  all  PP  must  he  awarded: 


and 


«A  =  2  M.i 

over 

contribution  divisions 

rA  =  21  (Mj  x  d-j) 

over 

contributinq  divisions 


(?)  Branch. 'T  The  S  x  Mj  x  d-j  productivity  points  distributed  to 
division  ,i  are  next  awarded  to  branches  of  this  division  which  contributed  to 
the  pro.iect.  If  branch  k  of  division  ,i  charned  M-j ^  man-months  to  the  pro.iect 
it  earns 


S  x  Mj  x  dj  x(Miic/Mi)x  bk 
or 

S  x  Mj|<  x  d-j  x  bk  productivity  points 
where  0. 7  _<  b|<  £  1.3 


^ In  the  h.reraahieal  distribution  o f  PP,  versons  at  the  dd '.vision  level  (not 
assi<jned  to  a  branch)  are  award  PP  at  t^is  level.  For  this  dr '  scussion, 
division  staff  personnel,  are  treated  the  sane  as  a  branch. 
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The  bk  i  •-  the  branch  iudonental  factor.  If  the  award  is  strictly  accordion  to 
manpower  "  ha  ropd ,  bk  =  ].  Since  all  divisional  ma::powar  charred  nr ’oi  nates 
fron  some  branch  and  all  PP  mis*  be  awarded: 


m  .  -  '•  M  ■ , 

’  ~  I  k 

O  v  e  r  1  p  n  t  r  i  b  i  j  t  i  n  r, 
h''an-~r:.,S  Of  d  ?  vi  S  ’  n*i  ' 

,jnH 

•*;  »  -  S'  v  d  i  X 

y m  r  "  of,  t  r  j  K  •  j  *-  i  o  n 

H  r  rj  n  r  h  p  <;  of  .  1 1  v  1  *■,  i  0  n  * 

'3)  Individual.  The  5  x  Mjk  x  *  •  x  bk  product i v'^v  points  distri¬ 
buted  to  branch  k  are  next  distributed  to  the  individuals  wi thin  « his  branch 
who  contributed  to  the  proiect.  If  individual  1  of  branch  k  of  division 
rharces  map -months  to  the  project ,  he  earns 

S  v  x  d.-  x  bk  x  'Mikl/Mik/  -x  Pi 
or 

S  x  M-jk]  x  dj  x  bk  x  pi  productivity  points 
where  0.7  _<  Pi  _<  1.3 

The  Pi  is  an  individual  judqmental  factor.  If  the  award  is  strictly  accordion 
to  manpower  charqed,  pi  =  1.  Since  all  branch  nanpower  orioi nates  from  some 
individual  and  all  PP  roust  be  distributed: 

Mik  =  1  Mi  k  1 

enntributinq 
i ndi vidual s 
of  branch  k 

Mik  x  d-j  x  bi,.  =  %  ( M j k i  x  d-j  x  bk  x  pi  ) 

enntributinq 
i ndividuals 
of  branch  k 


(4)  fli  scretionary  Productivity  Points  (DPP).  In  addition  to  the 
productivity  points  qenerated  by  the  project  score  and  nanpower  expended, 
there  is  another  class  of  points  -  discretionary  productivity  points.  These 
points  are  awarded  to  units  and  individuals  whose  contributions  to  a  project 
are  not  adequately  rewarded  by  the  standard  point  distribution  process. 
TRASANA  Meno  5-4  describes  DPP  and  the  constraints  on  their  awardinq. 
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c.  Assiqninq  the  Productivity  Index  (PI) 


The  PI  is  an  averaqe  of  project  PP  weiqhted  by  the  manpower  expended  on  the 
projects.  This  assiqnment  is  described  in  detail  for  divisions,  branches,  and 
i ndi vidua! s. 

(1)  Division.  If  division  j  contributes  to  n  projects  (includinq 
overhead)  for  i  =  1,  ?,  3,....,  n  with  scores  S-j  ,  expendinq  M-j  -j  man-months, 
earninq  DPP  4  then 
i  j 

21  (Si  X  Mi;j  x  d-jj)  +  DPP-j  i 
PI  =  1 _ ‘ _ _ 

f  Mij 

nPPi.i 


^?*Mii  ''  ^*Mij 


(?)  Rranch.  If  branch  k  of  division  j  contributes  to  n  projects 
(includinq  overhead)  for  i  =  I,  ?,...,  n  with  scores  Sj ,  expendinq  ^ 
man-months,  earninq  DPP-'’  then 

ijk 


2  (S.  x  M.  x  d.  .  x  b..  )  +  DPP.  .. 
i  1  i,lk  i.l  ik'  i.lk 


PI 


2  Mi;j 


.ik 


=  2  Sj  x  d i  i  x  bi 


k  x  j 


/Mi  jk  2 


DPP 


i  jk 


!  y  m  i  y  m. 

ijk/  ^  1,1k 


4  it  the  division  level  only  DPP  for  exceptional  contribution  and  project 
rescue  are  counted. 

'"’A t  the  branch  level  only  DPP  Dor  exceptional  contribution  and  project,  rescue 
are  counted.. 


1 


(3)  Individual.^  If  individual  1  of  branch  k.  and  division  j 
contributes  to  n  projects  (now  overhead  may  be  deleteo)  i  -  1,2,...,  n  with 
scores  ,  expending  M-j  j  ^  ]  man-months,  earning  D  F  P  -j  j  K  i  then 


PI 


1"  (S.  x  M.  .  x  d.  .  x  b..  *  Pil  )  +  D‘>P.  ..  , 
■'  V  1  Ukl  U  lk  1 J  K 1 


=  (S-j  x  x  d i  i  X  b -j  k  x  p i 1  x 


-  RP^iiki 

i 

'  -  M. . ,  , 

^  i  1  K  I 


3.  MEASUREMENT  ALGORITHM  ERROR  ANALYSIS 

Any  measurement  contains  some  error.  The  usefulness  of  the  measurement  can 
only  be  fully  realized  when  the  magnitude  of  the  possible  error  is  clearly 
specified.  This  analysis  provides  approximate  bounds  for  the  numerical  error 
associated  with  the  project  score  and  the  PI.  Numerical  error  in  the  computed 
values  of  a  function  is  dependent  upon  the  algorithm  chosen  for  the  computa¬ 
tion.  Once  a  correct  algorithm  is  chosen,  numerical  error  may  be  caused  by 
(1)  error  in  input  values,  (2)  algorithmic  error  (i.e.  round-off  error),  and 
(3)  mistakes  (i.e.  arithmetical).  The  project  scoring  sheets  indicated  the 
use  of  fairly  consistent  procedures  for  computing  project  scores  from  the 
input  variables.  The  analysis  of  numerical  error  in  project  scores  assumes 
these  procedures.  7  Standard  forms  are  used  to  compute  the  PI  given  the 
project  scores.  The  computational  procedure  suggested  by  these  forms  will 
also  be  assumed  for  the  analysis  of  the  error  in  the  PI. 

a.  Project  Score  Error 

First  the  error  in  each  of  the  four  factors  is  discussed,  and  then  total  error 
is  described. 

(1)  Quality.  The  value  of  Q  is  given  by  the  PRB  to  two  decimal 
places,  resulting  in  an  absolute  error  of  no  more  than  0.005.  Because  the 
value  of  Q  is  near  1,  the  maximum  absolute  relative  error  can  be  taken  to  be 
this  same  value  (0.005). 


I 


° The  PI  for  division  level  staff  is  computed  by  a  different  formula.  Refer  to 
TRASANA  Memo  .5 -4. 

? See  Conte  and  de  Boor  (1980)  for  a  discussion  of  numerical  error  analysis. 
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(2)  Manpower  utilization  efficiency. 

(a)  Project  scoring  sheets  indicate  that  the  formula  for  computing  R 


0.7 


when  Rp/Rft  <0.5 


R  = 


R  -  R 

1  -  °.3  A  P  when  0.5  _<  Rp/Ra  <1.0 

V  Rr,  / 


R  -  R  \ 

1  +  0.2  _P _ A  :  when  1.0  <  Rp/Ra  <  2.0 


\  0.5  Rr,  j 


1.2 


when  2.0^  Rp/Ra 


This  interpolation  formula  is  inconsistent  with  the  previous  definition  of  R  in 
terms  of  Rp/Ra-  In  fact,  the  above  formula  does  not  give  R  as  a  piecewise 


when  Rp/Ra  <  0.5 

1 

when  0.5  <^  RP/Ra  <  1.0 

1 

when  1.0  <  RP/Ra  <2.0 
when  2.0  j<  Rp/Ra 


(b)  Assuming  error  free  input  values,  the  use  of  the  interpolation 
formula  rather  than  the  formula  of  paragraph  2a (3)  could  cause  the  value  of  R  to 
be  in  error  by  as  much  as  0.05.  Since  the  value  of  R  is  rounded  to  two  decimal 
places  before  recording  on  the  project  scoring  sheet  and  inclusion  into  the 
multiplicative  formula  for  S,  the  absolute  maximum  error  could  be  as  much  as 
0.055.  Since  R  is  close  to  1,  0.055  would  also  be  an  approximation  of  the 
maximum  absolute  relative  error. 

(c)  A  more  useful  analysis  of  the  error  in  R  would  assume  the  use  of 
the  formula  of  paragraph  2a(3)  for  the  computation.  The  value  of  Rp  could  be 
assumed  to  be  exact,  so  that  input  error  is  entirely  the  result  of  Ra»  The 
value  of  Ra  (in  man-months)  to  two  decimal  places  is  provided  by  the  TRASANA 
f-'anpower  Utilization  Reporting  System  (TMURS).  <9  The  maximum  absolute  input 


linear  function  of  Rp/Ra.  but  rather 


0.7 


R  = 


1.3  -  0.3  (Rp/Ra') 

1.4  -  0.4  (Rp/Ra) 

1.2 


&The  most  recent  version  of  TMURS  reports  man-months  to  3  decimal  places . 


Prror  of  0.005  1  r  p  P  V  >  3  M  S  IP  *  U  ‘  a  --a  y  ’  "mj":  -< 1 .  <5, ' ,  1  orr  ,r  of  0.001  in 

.  Thus  0.0  "1]  approximates  the  na  *  VTm  abs.-,]  -it  r.  rp!  at  i  ye  ■‘"n.:r  in  ,  s  i  rrp 
i  s  nPH  r  1 . 


(d)  Trio  [WPS  manpower  ao"Oiinf  i^n  s  v  s  t  «-•••  ,  .-mpi  1  ps  ,  end  reports 

manpower  char  opt.  for  a!  1  typos  of  ac  f  i  v  i  -  i  ■- .  ”.c  ..v.^  -har  ips  a-,,  por.-ir  Ip  1  i  n 

mar-hour1:  and  report  ed  in  hot.h  c :  a:  ■•.  _  to : r  s  a».-'  .hi.-  m*  n<-. .  ;,ov/»*y.'*n  .  *  nr-  *  •»«. 

s  i  00  fro:-  .-'an-hoi.r  •;  1.0  na  n-mon  t  u  s  var  i  “>  hv  "lord  h.  'hio.  'i  r  -  -n-'t  h  ;  t  »■  ■"  ■ 

*■'  y  '  S  •  o*  i  f’vpil  i;;i  ’  ‘  nf  mp ‘  r  •  :  ■■•no  >n  v.  ■  ■  ■  •  ‘  . 

’"phr  if'r',' ,  i'  ir-  nn-'s,  and  one  pio  -  r ■  -./nr-  irq  p  ».  ’•  '  •  .  :/  . 

•nan.hni  .  -nnnpt.P'i  a  S  havin'’  V.ur'.  »  1  ]  ••  »t.  '  *  '  r  ;  •  i  '  1  :  •  ■■■* 

*  ••  <un  '  n  V  : .  i  c  r  o  u rl  ’  w'jf’"  !>p  u;  ■'(»-:>  m;  i  r;,,!  .  ■  yii  iw'c'  n/r>r_r  ins  'n  ;r> 

';'rnu  *•  if”’  P ,  a  f  i  xeo  .mi*,  shr-d  d  he  used  r  »r  ••••asurino  ho“n  :'n  a”  i 
’> .» ,  *h  ’  s  L  >’  ■••,iP-ho,.irs  ,  rao-wee ks  f  AO  ■’•an-ho  :r\'  ,  or  •.-io-mo”*'  r  s  wd  *  ••  c 

r'.'Vin  *;  5  .->";;al  f  a  Standard  "iri.pr  of  hours  Sljch  as  1  ;>P  ’’ia  r  - a  ; r  s . 


(3)  Computer  util  izet ion  efficiency.  nur  inq  the  first  scorinn  p«;rio-! 
nost  proiects  were  scored  with  an  exact  f.  value  nf  1.  Since  this  will  not  e 
thp  case  for  subsequent  soorino  periods,  the  numerical  errnr  due  to  variable  ■ 
is  described.  This  description  assumes  the  use  of  the  formula  of  naranraph 
?a(4)  and  that  the  value  of  Cp  is  exact.  Thus  all  input,  error  is  a  consequence 
of  C/\.  The  value  of  C/\  is  measured  in  Standard  Units  of  Processinq  (SUP)  wh’ch 
consists  of  the  sum  of  Central  Procession  Unit  (CPU)  time,  Innut/Output  (I/O) 
time,  and  Control  Card/Fxecuti ve  Request  (CC/ER)  time  in  hours.  Suppose  that 
the  vain®  of  C/\  is  reported  in  SUP  hours  to  the  nearest  hundredth  (3.5  seconds). 
This  error  in  C/\  results  in  an  approximate  maximum  absolute  error  of  O.nC]  in  1;. 
Pecause  C  is  near  1,  n.Ofll  approximates  the  maximum  absolute  relative  error  in 
C. 

M)  Timpliness.  The  timeliness  factor  is  a  function  of  the  devia¬ 
tion  of  the  actual  delivery  date  of  the  completed  pro.iect  report  from  the 
planned  delivery  date.  This  functional  relationship  is  proiect  dependent, 
detailed  error  analysis  would  require  this  function  in  addition  to  knowledne  id 
the  accuracy  with  which  the  deviation  in  dates  is  measured  (p.n.  nearest  Hay, 
nearest  week,  etc.).  If  the  deviation  can  he  assumed  to  he  exact  (in  davs,  sav) 
the  numerical  error  in  T  is  solely  a  consequence  of  the  current,  practice  of 
roundino  T  to  two  decimal  places  before  its  incorporation  into  the  formula  for 
S.  The  maximum  absolute  relative  error  is  then  0.005. 

(5)  Pro.iect  Score.  The  relative  error  in  S  is  simply  the  sum  of  the 
relative  errors  of  its  factors.  Since  S  is  near  1,  relative  error  approxi¬ 
mates  actual  error.  For  the  first,  scorinq  period  (usinq  the  interpolation 
formula  for  R)  the  project  score  has  maximum  absolute  (relative)  error  of 
approximately  0.071.  This  result  assumes  pxact  arithmetic  in  the  computations 
except  for  the  intermediate  roundinqs  of  R,  C,  and  T.  A  final  roundino  to  two 
decimal  places  results  in  an  approximate  error  of  0.075  in  the  project  score. 

A  more  useful  result  would  assume  that  the  formula  of  paragraph  ?a(3)  is  used 
to  compute  R.  If  the  value  of  T  is  assumed  to  be  exact  and  all  arithmetical 
computations  are  exact,  the  maximum  absolute  (relative)  error  in  S  is  approxi¬ 
mately  0.007.  If  intermediate  roundinos  of  R,  C,  and  T,  a  final  roundinq  of  S 
are  done  tn  two  decimal  places,  the  maximum  numerical  error  in  S  is  approxi¬ 
mately  0.027.  Rounds  for  the  numerical  error  in  the  project  score  are 
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summarized  in  table  F-2.  Thp  results  presented  there  assume  exact  arithmetic 
except,  where  otherwise  noted.  A  numerical  example  is  qiven  in  table  F-3  to 
illustrate  the  efficiency  of  the  bound  (0.0?7)  from  the  last  row  and  last 
column  of  table  F-2.  In  table  F-3  the  valup  of  S  qiven  in  the  first  column 
and  the  rounded  value  of  S  niven  in  the  last  column  differ  by  almost  0.0?*# 
thus  demonstrati nn  the  efficiency  of  the  n.n?7  bound.  A  reported  project 
score  could  then  be  in  error  hv  as  much  as  plus  or  minus  0.027.  For  example, 
if  a  reported  project  score  is  1.5*0  (base  case)  the  true  value  could  vary 
anywhere  from  1.5P7  to  1.533. 
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■  SUMMARY  DF  MIIMFRICAI.  FRRORS  ( ApPRflX ™ATF  MAXIMUM  ARSOLIITF  FRRDR) 


VARIARLF 

Interpolation  Formula  for  R 

! 

Formula  of  2a(3)  for  R 

NO  INTFRMFOIATF 
R0UN0ING 

INTFRMFOIATF 

POtINOINfi 

INTFRMFOIATF | 
R  01 1  NO  I  NR 

n 

hhhhumni 

R 

.OR 

.055 

.001 

i 

.006 

0 

.001 

.006 

.001 

.006 

T 

NA 

.005 

NA 

.005 

S 

.ORfi 

.071 

.007 

.022 

S  R0IIN0F0 
TO  TWO 
OFCIMAL 
PLACES 

.0*1 

.076 

.012 

.027 

h.  Productivity  Index  Frror 

Followinn  the  formulas  of  2c  the  PI  may  be  computed  bv  summinq  PP  and  manpower 
expended  and  dividinq  the  two.  Msinq  exact  arithmetic,  the  maximum  absolute 
error  in  a  computed  PI  is  bounded  by  the  maximum  absolute  error  in  the  project 
scores  plus  a  much  smaller  term  involvinq  the  number  of  projects,  total 
manpower,  and  total  PP.  In  fact,  the  maximum  absolute  error  (0.027)  in  the 
project  score  alone  is  a  realistic  approximation  of  the  numerical  error  in  the 
PI.  Thus  a  reported  PI  could  be  in  error  by  as  much  as  plus  or  minus  0.027. 

In  the  first,  scorinq  year  there  was  only  a  difference  of  0.03  between  the 
lowest  PI  of  the  top  nine  scorinq  individuals  and  the  PI  of  the  next  hiqhest 
scorer.  This  difference  could  in  fact  then  be  due  almost  entirely  to 
numerical  error. 
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4.  RELATIONSHIPS  AMONG  THE  PROJECT  SCOPE  VARIABLES 

A  traditional  measure  of  productivity  is  value  of  output  divided  by  value  of 
input.  The  project  scorinq  function  which  is  a  qlohal  measure  of  a  project 
team's  productivity  is  a  variation  of  this  traditional  measure.  If  f  and  o 
denote  the  piecewise  linear  functions  of  paraqraphs  ?a(3)  and  (4)  respectively 
then 


S  =  PxOxTxf  (  Rp/Ra')  x  q  (  Cp/Cp) 


In  a  sense,  Rp  and  Cp  represent  the  output  values  of  a  project  (as  determined 
by  the  PMR)  and  Ra  and  Cp  represent  input  values.  Rather  than  use  ratios 
directly,  they  are  adjusted  by  f  and  q  to  take  into  account  the  relative  costs 
of  these  two  resources.  The  0  and  T  (and  also  P)  factors  miqht  be  viewed  as 
"value  multipliers."  Timeliness  or  quality  increase  the  worth  of  a  project,  o 
perhaps  more  real istical ly,  the  lack  of  timeliness  or  quality  decrease  its 
orth. 


a.  Schemes  for  Assembling  Factors 


The  method  for  assembling  the  individual  factors  into  a  project  score,  the 
project  scoring  function,  is  a  combined  multiplicative  and  conjunctive  scheme. 
To  contrast  alternate  scoring  schemes,  suppose  that  the  project  score  S  is  a 
function  of  Q  and  R  alone. 

(1)  An  additive  or  compensatory  scheme  would  compute  S  as  S  =  Q  +  R. 
Using  Q  =  1.3  and  R  =  1.0  as  base  values  for  a  B  or  "good"  project,  the 
category  of  B  or  better  projects  is  defined  by  S  2  2.3.  The  compensatory 
property  of  this  scheme  is  illustrated  by  the  fact  that  a  project  with  Q  =  1.2 
and  R  =  1.1  and  a  project  with  0  =  1.4  and  R  =  0.9  both  score  2.3. 

(2)  A  conjunctive  scheme  would  define  a  B  or  better  project  as  one 
with  Q  >  1.3  and  R  2  1.0.  Now  a  low  score  on  one  factor  can  not  be  compensated 
for  by  a  high  score  on  another  factor. 

(3)  A  strictly  multiplicative  scheme  for  S  would  compute  S  by 
S  =  Q  x  R.  A  project  graded  8  or  better  would  be  defined  as  one  with 

S  2  1.3  =  (1.3)  (1.0).  To  some  extent  this  scoring  function  is  also  compensa¬ 
tory.  For  example,  a  project  with  Q  =  1.35  and  R  =  0.962  and  a  project  with 
Q  =  1.2  and  R  =  1.0833  both  have  S  =  1.3.  Thus  a  low  value  on  one  factor  can 
be  compensated  for  by  a  higher  value  on  another. 

(4)  Although  the  TPMS  scoring  function  is  multiplicative,  to  avoid 
the  compensatory  property  (at  least  for  Q)  a  minimum  acceptable  value  of 

Q  =  1.3  must  be  achieved.  In  theory  a  project  scoring  less  than  1.3  on  Q  would 
be  required  to  expend  more  manpower  to  upgrade  quality  to  at  least  1.3.  In 
computing  S  the  original  Q  value  is  used  with  the  inflated  R  value.  In  prac¬ 
tice,  because  some  R  scores  are  so  high,  low  scores  for  Q  are  in  effect  compen¬ 
sated  for.  In  the  first  scoring  period  for  example,  one  project  had  Q  =  1.2 
and  R  =  1.122  for  S  =  1.3464. 9  Another  project  had  0  =  1.35  and  R  =  .998  for 
S  =  1.3474.  Thus  it  appears  that  a  low  score  on  0  is  compensated  for  by  a  high 
score  on  R  and  vice  versa. 

(5)  Figures  E-4  to  E-7  illustrate  the  different  scoring  schemes  just 
discussed.  The  shaded  area  in  each  corresponds  to  scores  of  B  or  better. 

b.  Distinct  Resource  Efficiency  Factors 

Two  resource  efficiency  factors  are  used  to  compute  the  project  score.  Two 
difficulties  arise  as  a  consequence  of  these  factors. 

(1)  Scale  factors.  After  the  ratio  of  planned  resource  to  actual 
resource  is  computed,  it  is  scaled  by  a  piecewise  linear  function  before  it  is 
factored  into  the  project  score.  These  scale  functions  (f  and  g  in  the 
notation  of  the  previous  section)  are  the  source  of  one  difficulty. 


&For  illustration  only,  S  =  Q  x  R 


E- 16 


!-d)  If 

a  :np 

fO 

out  res  c  ID 

i  n 

an 

n  owp  r 

.;sa 

oe 

cl n  • 

'  ]  t.  *« 

s  ac 

conn  i 

i  s'lPd 

Vi 

i  t 

n 

••  V’i 

»Q  •'  *  h  ^  *■ 

he  rat 

1  0 

i‘  f  c  i  r  i  p  p  o  v 

1  s 

? 

, 

The 

toms 

sc 

•if- 

-  f  :  n  c 

f  ’  o  n 

WO  i  i  1 

H 

a 

ss 

i  < 

in  a 

■vmrowp 

r  ,i  f  (  ] 

cienev  f  a C  4  o r 

cf 

1 

#  ? 

.  If 

a  i  ■ 

r~,  K 

r*p 

’ 1  i  ’’  r  •  •  s 

c  i 

'I  r  .; 

O-npi  i  *■  p 

■;  a 

rjT 

'  an< 

1  i  *■  is 

done  vi 

i  t  h 

t  £  ,*/ri  r  *  h  t 

f  hp 

ra 

*.io  P 

ff  ic 

'  PDl  V 

i  -  al 

S  ) 

^  n 

T'  ;v.« f 

S 

-a 

1  ( 

■»  fi  ;f 

■c  4  i  ou  w 

■  iuM  a 

ss 

nn  a  r  r;* 

pr 

P 

( f 

k  i  p  n 

c  V  f 

a  n t_ 

,-r 

o  f  1  . 

1 . 

The  r 

at'  i  p - 

n 

al 

P 

f  n  r 

niff  pro 

n  t  s  c  a 

1  o 

functions  i 

s  r 

t.  n  p 

ah  so 

1  !.  4 1 

p  ■; 

:onp‘i  •. 

P r  -~ 

i  !  s r  s 

a  r  p 

r 

ri.-i 

1  1 

!  nor 

inarm  > 

o  an  so 

]  i :  t 

p  ’  iti  rs  pov/p  r 

<  )  s 

t 

W  1 

‘  h 

*  tap  rr 

*'»  ;v  ’  V 

■i  ■  i 

n 

/  1  Tf 

;  svsf‘M-1 

t  ho  a 

f.  i 

7  1  f  V  ha  S  ^  • 

* 

. 

*  i  .  r  ■  I 

1  Ci  i 

apau 

f. 

J  0 

r  , 

d-  i 

'*  r-'O 

;  r  , 

’’  s 

n  *  ;  !  1  V 

« 

f  .  i  c 

•a. 

/  .N- 

t  a 

p  ■’  1 

:  '  ;* 

L 

n  r  a  ‘i 

a  r 

;  >  i  ’  I  '  ■  ;  ‘ 

...  .  .  -j  r 

-»  f 

j 

1 

i  r  •  ■  < 

-  ■  - 

■  .  *  ■  i  <i  '  ""  n  r’ 1 ; 

r,;o 

*■  Ir,  1- 

■  V  C  i 

’ 

*-  ;co 

o  ;  U  •; 

• 

*  i  •  •  •  • 

- 

.  ‘  1  .  -1  * 

"  •  rf  T  ; '  )  //f  ‘ 

r 

M 

r  f»,' 

*  i :  .c- , 

:  r) 

• 

o  '  ' 

.  ,  c 

i  "> 1  If  fdp  activity  believes  that  it  has  sum!  us  c  cm\ic  i  *  v 

and  that  *  t  shouH  ercouraoe  mu-  use  of  f.np  computer,  it  imow*.  *-n  Kor/  m 
eliminate  the  computer  factor  fro  n  tl’P  scorinn  tiincf  ion  al tooether , 

(?)  nistinct  Factors.  Another  rational  for  navioo  different  avate 
functions  is  that  it  mi qht.  encourage  trad  inn  relatively  cheap  computer  ;:s>-  for 
more  expensive  manpower.  However,  because  the  project  score  is  a  product  'f 
distinct  factors,  in  neneral  the  optimal  procedure  is  to  maximize  ear1-'  fac4oi 
independent.lv  of  the  other.  In  fact,  it  is  not.  hard  to  construct  examples 
where  choan  computer  usaoe  mi  qht  he  replaced  hv  expensive  manpower  in.  order  t 
increase  the  project  score.  For  example,  suppose  a  project,  is  initiated  with 
Op  =  ?n  and  C.p  -  ?,  Suppose  that  well  into  the  study,  the  study  manaqer  is 
confronted  with  the  possibility  of  completion  the  project  with  P/\  =  IS  and 
f-A  =  ?.  In  this  case  R  x  (I  =  1.07.  Havinq  already  tispd  1  SHP,  he  does  have 
the  option  of  doino  the  1  SUP  of  computer  work  renaininn  with  S  man-months  of 
manpower.  If  he  chooses  this  option  the  project,  will  he  completed  witn 
P^  =  ?f>  and  f.A  -  1.  In  this  case  R  x  C  =  1.1.  If  all  other  factors  are  the 
same,  replacino  computer  usaqe  hv  manpower  would  result  in  a  hiqher  project 
score.  If  it.  is  desired  that  computer  use  replace  manpower  when  it  is  cost 
elective,  realistic  costs  for  each  resource  should  be  oiven  and  the  project, 
s-ore  function  should  use  a  sinnle  factor  which  includes  both  manpower  and 
computer  use  in  relation  to  their  costs. 

c.  The  Differential  of  S 

The  differential  (dS)  of  S  is  useful  in  assessinq  effects  of  chanqes  in  the 
values  of  the  variables  on  the  value  of  S.  In  neneral,  the  differential  of  a 
function  can  be  used  to  approximate  chanoes  in  function  values  oiven  chanqes 
in  its  independent  variables.  In  this  case  the  differential  is 


dS  =  P  r TR r  u  0  +  OPT  AT  +  OTC  AP  +  OTR  AC] 

Recause  D  is  scaled  with  lamer  values  than  T,  R,  or  C;  small  chanqes  (AC , AT , 
&R)  in  these  variables  will  have  a  oreater  effect  on  the  value  of  S  than  will 
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a  small  chanqe  [lO)  in  0.1 n  Table  F-4  illustrates  the  effect  of  incrementinq 
K  the  variables  by  n.l.  Table  F-5  does  the  sane  for  an  increment  of  0.005. 

Notice  that  a  smaller  value  of  S  results  from  incrementinq  0.  Scalinq  0  with 
larqer  values  has  an  effect  of  emphasizinq  the  other  variables  in  the  formula 
for  S. 


TABLE  F-4 

THF  EFFECT  ON  THF  BASF  SCOPE  OF  INCREMENTING 
VARIABLES  BY  n.l 


CASE 

0 

- T 

R 

c 

S 

BASF 

1.3 

1 

1 

1 

1.550 

A  OIIALITY 

1.4 

1 

1 

1 

1.580 

REST  TIME 

1.3 

1.1 

1 

1 

1.716 

Rp=1.5  R/\ 

1.3 

1 

1.1 

1 

1.716 

Cp=?CA 

1.3 

1 

1 

1.1 

1.7  16 

TAR1.F  E-5 

THE  EFFECT  ON  THE  BASF  SCORF  OF  INCREMENT  I  NO 
VARIABLES  BY  0.05 


|  CASE 

0 

T 

R 

C 

S 

|  RASE 

1.3 

1 

1 

1 

1.550 

R+  OIIALITY 

1.35 

1 

1 

1 

i.fi?n 

noon  time 

1.3 

1.05 

1 

1 

1.538 

Rp=l.?5  Ra 

1.3 

1 

1.05 

1 

1.538 

Cp=l.bCA 

1.3 

1 

1 

1.05 

1.538 

IQ  In  the  first  Rearing  period  the  smallest  attained  value  of  0  (1.13)  was 
areater  than  maximum  possible  values  ? o r  T  (1.1)  or  C  (1.1).  Also  for  al 
except  3  projects  the  value  of  0  was  areater  than  or  eaual  tn  the  maximum 
value  of  R. 
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s. 

The  project  score  is  a  nlohal  measure  nf  the  protect  team's  relative  prndur- 
r  ivit.v.  The  mechan i  s,n  for  awa Hinn  oreater  or  fewer  op  t  nan  that  a  •  lot  *■  <*  K  *• 
straiqht  charge  to  manpower  award  allows  a  nanaoer  to  -li  * ferer*  iate  amo'.u  *-ii 
quality  of  individuals'  or  units'  r  on  f.M  notions, 

a.  I  r  rhe  hierarchical  process  nf  awar-Mno  of  a  f  ;du"i--nt a '  far  '  >>r 

l  i  1 1 0  or  ‘  a  C  'r‘  s  *  e  t'l  Cie  the  process  f  • 1  ■*  /  i  S  '■  on  n  i.  '  r.H  '  l' 

ri  *■  a  1  f  a  r *-  fi  r  .  h  n  r n  nrj  r  nQ;-.  f’.  ^  7  J  *  1  %  S  u .  ,  r,*in  U  n  ■ ;  (  . 1 1  *  h ,  •  - 1 ;  j  1  :  ;  r,  * 

*•  r  i  buf !  of  tnat  on  t.  The  factor  equal  s  *  ’  f  *i-p  pn  i  -  ►  <{i  str’O  , 

e  .•!  1  r  d  stri(  f.  1  v  accordion  to  'nan  power  isaie.  A*  +,e  '  ,  u-  i  •>  i 

st  r<.  • ohf.  ;harop  to  manpower  a  war  a  o'u’  <t  be  increase  '  *>y  ■<«  n„rt-.  ,  . 

f.v  :  ir  -.f  nr  decreased  hy  as  "  uch  as  factor  n*  . 

h  .  the  hierarchical  a  i  St  r  1  on  i  on  pf  dudomen*  a  1  PP  ana  compound  i  *>  i  ef  *p 

of  nil1 1  i  pi  y  i  no  hy  the  iudqnental  factors  nioht  result  in  unintended  results. 
Consider  the  example  illustrated  in  tahle  F-h.  Suppose  it  is  the  juannent  ) 

t. hp  study  manaqer  and  division  chief  that  persons  P  and  S  did  equally  haclv 

and  should  receive  no  more  than  the  absolute  minimum  awarq.  Person  1  r.arrie 
the  p’-oiect  and  should  receive  the  maximum  possible  award.  As  shown  in  the 
tahle,  since  persons  P  and  3  helonn  to  different  branches,  person  ?  automat i 

callv  receives  more  PP  and  a  hiqher  PI. 


6.  WEIGHTED  AVERAGE 

The  PI  is  simply  a  weiqhted  averaqe  over  a  scorinq  period.  Specifically  it  is 
an  averaqe  of  project.  scores  weiohted  bv  their  manpower  requirements.  Here 
overhead  is  considered  as  a  project  with  a  score  of  1.  Because  however,  a 
non-overhead  project  could  carry  over  from  one  period  to  the  next  while 
overhead  would  not,  spurious  results  can  he  obtained.  The  data  in  table  E-7 
illustrates  just  this.  The  branches  seem  to  be  equally  productive  durinq  the 
two  years,  however  this  is  not  reflected  in  their  Pi's. 


TARI.F  E-7 

AN  EXAMPLE  OF  COMPUTING  BRANCH  PI'S  AS  WEIGHTED  AVERAGES 


1ST  PERIOD 

2ND  PERIOD 

1  TMM  Overhead 

1  TMM  OVERHEAD 

2  TMM  on  Project  with  S  =  l.S 

BRANCH  A 

Q  TMM  carried  over . 

11  TMM  on  project  with  S  =  l.S 

total  TMM  =  3 

total  TMM  =  21 

total  PP  =  4 

total  PP  =  31 

PI  =  4/3  =  1.333 

PI  =  31/21  =  1. 476 

1  TMM  Overhead 

1  TMM  Overhead 

RPANCH  R 

11  TMM  on  project  with  S  =  l.S 

11  TMM  on  project  with  S  =  l.S 

total  TMM  =  12 

total  TMM  =  12 

total  PP  =  17. S 

total  PP  =  17. S 

17. S 

PI  =  =  1.458 

17. S 

PI  =  =  1.4S8 

12 

12 
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AN  ALTERNATIVE  TPMS  SCORING  PROCEDURE 


1.  GENERAL 

The  purpose  of  this  study  was  to  evaluate  the  first  year  functioning  of  TPMS; 
however,  possible  system  improvements  which  followed  directly  from  the 
evaluation  were  to  be  offered.  This  appendix  presents  a  conceptual  scoring 
alternative  which  has  as  its  basis  improving  system  functioning  in  light  of 
the  original  system  design  attributes.  This  alternative  is  simple,  would 
improve  the  systan  flexibility,  and  should  considerably  lessen  resistance  to 
TPMS  because  it  specifically  addresses  those  issues  most  frequently  cited  as 
problems  with  the  current  system. 

2.  SCORING  FACTORS 

The  alternative  procedure  retains  the  manpower  utilization  efficiency  factor 
(R),  the  quality  factor  (Q),  and  the  timeliness  factor  (T).  A  distinct 
computer  utilization  efficiency  factor  is  not  used.  As  long  as  the  computer 
remains  a  somewhat  unlimited  resource  with  fixed  cost,  and  manpower  costs 
account  for  approximately  80  percent  of  TRASANA  resources,  a  separate  computer 
factor  is  not  seen  as  cost  effective.  If  the  magnitude  of  computer  use  and 
costs  should  change  in  the  future,  computer  usage  could  be  incorporated  into 
the  scoring  procedure.  If  this  is  done  however,  a  more  appropriate  method 
might  be  to  have  a  combined  resource  efficiency  factor  which  incorporates 
manpower  and  computer  usage.  This  combined  resource  efficiency  factor  could 
then  be  weighted  to  reflect  the  relative  cost  of  the  two  resources.  Each  of 
the  three  factors  is  discussed  separately. 

a.  Manpower  Utilization  Efficiency  Factor  (R) 

Figure  F-l  is  the  graph  of  a  generic  efficiency  factor  Rp/RA  as  a  function  of 
RA/Rp.  Figure  F-2  is  the  current  efficiency  factor  R,  also  as  a  function  of 
RA/Rp.  The  alternative  suggested  for  scoring  R  is  in  Figure  F-3.  The  verti¬ 
cal  scaling  is  just' ordi nal  ,  i.e.,  simply  three  categories  of  efficiency.  A 
project  with  actual  manpower  within  plus  or  minus  10  percent  of  planned  man¬ 
power  would  be  placed  in  the  medium  category.  When  actual  manpower  is  10 
percent  over  planned  manpower,  it  is  placed  in  the  low  category.  Projects 
with  actual  manpower  of  less  than  90  percent  of  planned  manpower  are  placed  in 
the  high  category.  In  addition,  a  division  level  check  on  R  should  be  imple¬ 
mented.  For  example  a  division  might  be  required  to  have  a  given  distribution 
of  projects  among  the  three  categories,  or  bonus  points  could  be  awarded  for 
achieving  a  relatively  symmetrical  distribution  with  over  half  of  the  projects 
in  the  middle  category.  Another  possibility  is  to  give  rewards  or  bonus 
points  to  divisions  whose  average  Rp/RA  is  near  1. 


Figure  F-3.  Alternative  Manpower  Efficiency  Factor  (R) 


b.  Qua! i ty 


The  alternative  scoring  procedure  requires  that  quality  also  be  ordinal  ly 
scaled  with  three  categories:  high  (excellent),  medium  (okay),  low  (substan¬ 
tial  revision  required  to  release).  This  categorical  scoring  would  produce 
the  desired  discrimination  among  projects  and  would  be  much  more  realistic  in 
terms  of  expectations  from  PRB  members.  The  potential  for  placing  a  project, 
into  the  wrong  category  would  be  considerably  less  t:  an  the  potential  error  in 
the  current  grading  scheme. 

c.  Timeliness 


This  factor  would  also  be  scaled  ordinally  with  the  following  three  cate¬ 
gories:  high  (early),  medium  (on  time),  low  (late).  Figure  F-4  shows  the 
graph  for  this  factor. 

3.  COMBINING  THE  FACTORS 

a.  The  three  distinct  levels  or  categories  for  each  of  the  three  factors 
results  in  a  total  of  twenty-seven  distinct  possible  results  for  any  project. 
These  outcomes  are  represented  with  the  tree  diagram  in  Figure  F-5.  Ranking 
of  the  projects  on  a  productivity  scale  can  now  be  accomplished  by  sorting  the 
various  outcomes  according  to  the  importance  attached  to  the  level  of  each 
factor.  Numerical  values  could  then  be  assigned  to  the  sorted  categories. 

This  would  allow  for  weighted  (by  manpower)  averages  of  project  scores  to  be 
computed  for  organizational  units. 

b.  Table  F-l  gives  one  possible  sort  of  categories,  assuming  that  each 
factor  is  of  equal  importance  and  that  the  levels  across  factors  are 
comparable.  In  this  scoring  scheme  there  are  seven  distinct  categories.  If 
this  scoring  scheme  were  applied  to  the  first  year  results,  the  resulting 
distribution  is  shown  in  figure  F-6. 

4.  ALTERNATIVE  SCORING  PROCEDURE  FLEXIBILITY 

a.  In  addition  to  its  simplicity,  the  alternative  scoring  procedure  is 
extremely  flexible.  For  example,  if  overestimation  Rp  is  not  sufficiently 
discouraged,  the  computation  of  R  could  be  altered  to  include  a  penalty  window 
for  overestimation.  Figure  F-7  shows  two  methods  by  which  this  could  be  done. 
If  a  project  has  actual  manpower  between  50  and  90  percent  of  planned  man¬ 
power,  an  R  value  of  "medium"  could  be  assigned  (upper  dotted  line),  or  an  R 
value  of  "low"  could  be  assigned  (lower  dotted  line).  Using  the  latter 
choice,  the  projects  for  the  first  scoring  period  were  given  scores  by  the 
alternative  procedure.  The  results  are  shown  by  the  histogram  in  figure  F-8. 
This  histogram  can  be  compared  to  the  one  in  figure  F-6.  Notice  that  dis¬ 
couraging  or  penalizing  overestimating  results  in  a  more  symmetrical  distri¬ 
bution. 
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Figure  7-5.  Scoring  Outcomes 
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A  RANKING  OF  SCORING  CATEGORIES 


Outcome  Category 

7 
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Figure  F-6.  First  Year  Project  Scores  Alternative  Scoring  Procedure 


CATEGORY 


Figure  F-8.  Alternative  Scoring  Procedure  with 

Penalty  for  Overestimation  of  Manpower 
Applied  to  First  Year  Project  Scores 


b.  As  another  example  of  the  flexibility  of  the  alternative  system, 
suppose  that  it  is  decided  that  the  timeliness  factor  should  be  scored: 
high-on  time,  medium-late,  and  low-very  late.  The  levels  of  this  factor 
probably  should  not  be  treated  as  comparable  to  those  of  the  other  two,  and  a 
new  ranking  of  the  27  scoring  outcomes  of  figure  F-5  might  be  required.  Tabl 
F-2  gives  one  of  many  possible  ranking  schemes. 
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A  RANKING  OF  SCORING  CATEGORIES 


TIMELINESS  ALTERED 
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c.  The  use  of  three  levels  for  each  of  the  factors  seems  to  be  simple  to 
use  yet  provides  enough  discrimination  among  the  projects;  however,  more  or 
fewer  levels  might  also  be  considered.  For  example,  the  timeliness  factor 
might  be  altered  to  have  only  two  levels:  medium-on  time  and  low-late.  If 
the  other  factors  remain  the  same,  the  tree  diagram  of  figure  F-9  shows  the  18 
possible  outcomes  for  this  scheme  and  table  F-3  gives  one  possible  ranking  of 
the  outcomes. 
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Figure  F-9.  Scoring  Outcomes  with  Two  Levels  for  Timeliness 
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TABLE  F-3 


A  RANKING  OF  SCORING  CATEGORIES 
TWO  LEVELS  FOR  TIMELINESS 


CATEGORY  RANKING 

OUTCOME  CATEGORY 
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(H,  H,  M) 

5 

(H,  H,  L) 

(H,  M,  M) 
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(H,  L,  L)  (L,  H,  L) 
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2 

(M,  L,  L) 

(L,  M,  L) 

(L,  L,  M) 
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